2023-04-06 05:56:05,781 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099 2023-04-06 05:56:05,794 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-06 05:56:05,809 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-06 05:56:05,809 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962, deleteOnExit=true 2023-04-06 05:56:05,810 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-06 05:56:05,810 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/test.cache.data in system properties and HBase conf 2023-04-06 05:56:05,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/hadoop.tmp.dir in system properties and HBase conf 2023-04-06 05:56:05,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/hadoop.log.dir in system properties and HBase conf 2023-04-06 05:56:05,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-06 05:56:05,812 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-06 05:56:05,812 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-06 05:56:05,923 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-06 05:56:06,295 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-06 05:56:06,300 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-06 05:56:06,301 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-06 05:56:06,301 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-06 05:56:06,301 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 05:56:06,302 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-06 05:56:06,303 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-06 05:56:06,303 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-06 05:56:06,303 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 05:56:06,304 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-06 05:56:06,304 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/nfs.dump.dir in system properties and HBase conf 2023-04-06 05:56:06,305 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/java.io.tmpdir in system properties and HBase conf 2023-04-06 05:56:06,305 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-06 05:56:06,305 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-06 05:56:06,306 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-06 05:56:06,786 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 05:56:06,791 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 05:56:07,042 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-06 05:56:07,224 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-06 05:56:07,239 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 05:56:07,271 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 05:56:07,304 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/java.io.tmpdir/Jetty_localhost_localdomain_45297_hdfs____.s09qbb/webapp 2023-04-06 05:56:07,426 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45297 2023-04-06 05:56:07,464 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-06 05:56:07,465 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-06 05:56:07,859 WARN [Listener at localhost.localdomain/32913] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 05:56:07,957 WARN [Listener at localhost.localdomain/32913] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-06 05:56:07,972 WARN [Listener at localhost.localdomain/32913] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-06 05:56:07,977 INFO [Listener at localhost.localdomain/32913] log.Slf4jLog(67): jetty-6.1.26 2023-04-06 05:56:07,981 INFO [Listener at localhost.localdomain/32913] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/java.io.tmpdir/Jetty_localhost_32927_datanode____5ie0va/webapp 2023-04-06 05:56:08,064 INFO [Listener at localhost.localdomain/32913] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:32927 2023-04-06 05:56:08,395 WARN [Listener at localhost.localdomain/33939] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-06 05:56:08,759 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d3be14af335bac7: Processing first storage report for DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3 from datanode aceb380d-48cd-4fee-9534-415073b880c6 2023-04-06 05:56:08,760 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d3be14af335bac7: from storage DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3 node DatanodeRegistration(127.0.0.1:46105, datanodeUuid=aceb380d-48cd-4fee-9534-415073b880c6, infoPort=46315, infoSecurePort=0, ipcPort=33939, storageInfo=lv=-57;cid=testClusterID;nsid=1317408443;c=1680760566858), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-06 05:56:08,760 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8d3be14af335bac7: Processing first storage report for DS-3276415d-241b-4013-850c-d0965d089b4c from datanode aceb380d-48cd-4fee-9534-415073b880c6 2023-04-06 05:56:08,760 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8d3be14af335bac7: from storage DS-3276415d-241b-4013-850c-d0965d089b4c node DatanodeRegistration(127.0.0.1:46105, datanodeUuid=aceb380d-48cd-4fee-9534-415073b880c6, infoPort=46315, infoSecurePort=0, ipcPort=33939, storageInfo=lv=-57;cid=testClusterID;nsid=1317408443;c=1680760566858), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-06 05:56:08,927 DEBUG [Listener at localhost.localdomain/33939] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099 2023-04-06 05:56:08,997 INFO [Listener at localhost.localdomain/33939] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962/zookeeper_0, clientPort=55165, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-06 05:56:09,011 INFO [Listener at localhost.localdomain/33939] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55165 2023-04-06 05:56:09,020 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:09,022 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:09,688 INFO [Listener at localhost.localdomain/33939] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1 with version=8 2023-04-06 05:56:09,689 INFO [Listener at localhost.localdomain/33939] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/hbase-staging 2023-04-06 05:56:09,990 INFO [Listener at localhost.localdomain/33939] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-06 05:56:10,409 INFO [Listener at localhost.localdomain/33939] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 05:56:10,440 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,440 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,441 INFO [Listener at localhost.localdomain/33939] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 05:56:10,441 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,441 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 05:56:10,600 INFO [Listener at localhost.localdomain/33939] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 05:56:10,677 DEBUG [Listener at localhost.localdomain/33939] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-06 05:56:10,762 INFO [Listener at localhost.localdomain/33939] ipc.NettyRpcServer(120): Bind to /148.251.75.209:35907 2023-04-06 05:56:10,771 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:10,774 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:10,791 INFO [Listener at localhost.localdomain/33939] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35907 connecting to ZooKeeper ensemble=127.0.0.1:55165 2023-04-06 05:56:10,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:359070x0, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 05:56:10,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35907-0x10084d914bb0000 connected 2023-04-06 05:56:10,855 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 05:56:10,856 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 05:56:10,859 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 05:56:10,866 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35907 2023-04-06 05:56:10,866 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35907 2023-04-06 05:56:10,867 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35907 2023-04-06 05:56:10,867 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35907 2023-04-06 05:56:10,867 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35907 2023-04-06 05:56:10,873 INFO [Listener at localhost.localdomain/33939] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1, hbase.cluster.distributed=false 2023-04-06 05:56:10,950 INFO [Listener at localhost.localdomain/33939] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-06 05:56:10,951 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,951 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,951 INFO [Listener at localhost.localdomain/33939] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-06 05:56:10,951 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-06 05:56:10,952 INFO [Listener at localhost.localdomain/33939] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-06 05:56:10,957 INFO [Listener at localhost.localdomain/33939] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-06 05:56:10,960 INFO [Listener at localhost.localdomain/33939] ipc.NettyRpcServer(120): Bind to /148.251.75.209:33239 2023-04-06 05:56:10,962 INFO [Listener at localhost.localdomain/33939] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-06 05:56:10,973 DEBUG [Listener at localhost.localdomain/33939] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-06 05:56:10,974 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:10,978 INFO [Listener at localhost.localdomain/33939] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:10,981 INFO [Listener at localhost.localdomain/33939] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33239 connecting to ZooKeeper ensemble=127.0.0.1:55165 2023-04-06 05:56:10,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:332390x0, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-06 05:56:10,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33239-0x10084d914bb0001 connected 2023-04-06 05:56:10,986 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 05:56:10,987 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 05:56:10,988 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-06 05:56:10,990 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33239 2023-04-06 05:56:10,990 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33239 2023-04-06 05:56:10,991 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33239 2023-04-06 05:56:10,991 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33239 2023-04-06 05:56:10,991 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33239 2023-04-06 05:56:10,993 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,001 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 05:56:11,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 05:56:11,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-06 05:56:11,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,027 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 05:56:11,028 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,35907,1680760569837 from backup master directory 2023-04-06 05:56:11,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-06 05:56:11,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-06 05:56:11,032 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 05:56:11,032 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,035 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-06 05:56:11,036 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-06 05:56:11,134 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/hbase.id with ID: f9dc232b-22f9-451c-8836-8ef2290ef3ad 2023-04-06 05:56:11,182 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-06 05:56:11,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,236 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6e4f328e to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:11,274 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4249aef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:11,297 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:11,299 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-06 05:56:11,318 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-06 05:56:11,318 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-06 05:56:11,320 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:11,324 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:11,325 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 05:56:11,356 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store-tmp 2023-04-06 05:56:11,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:11,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 05:56:11,382 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:56:11,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:56:11,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 05:56:11,382 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:56:11,382 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:56:11,383 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 05:56:11,384 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/WALs/jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,405 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C35907%2C1680760569837, suffix=, logDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/WALs/jenkins-hbase20.apache.org,35907,1680760569837, archiveDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/oldWALs, maxLogs=10 2023-04-06 05:56:11,457 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK] 2023-04-06 05:56:11,464 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:11,525 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/WALs/jenkins-hbase20.apache.org,35907,1680760569837/jenkins-hbase20.apache.org%2C35907%2C1680760569837.1680760571414 2023-04-06 05:56:11,526 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK]] 2023-04-06 05:56:11,526 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:11,527 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:11,530 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,531 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,584 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,590 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-06 05:56:11,611 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-06 05:56:11,624 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:11,629 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,631 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,650 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-06 05:56:11,655 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:56:11,656 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68045751, jitterRate=0.013960704207420349}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 05:56:11,657 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 05:56:11,658 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-06 05:56:11,675 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-06 05:56:11,675 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-06 05:56:11,677 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-06 05:56:11,679 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-06 05:56:11,704 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-04-06 05:56:11,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-06 05:56:11,725 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-06 05:56:11,729 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-06 05:56:11,749 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-06 05:56:11,752 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-06 05:56:11,754 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-06 05:56:11,758 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-06 05:56:11,761 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-06 05:56:11,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,765 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-06 05:56:11,766 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-06 05:56:11,776 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-06 05:56:11,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 05:56:11,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-06 05:56:11,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,780 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,35907,1680760569837, sessionid=0x10084d914bb0000, setting cluster-up flag (Was=false) 2023-04-06 05:56:11,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,795 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-06 05:56:11,797 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,800 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:11,804 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-06 05:56:11,805 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:11,808 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.hbase-snapshot/.tmp 2023-04-06 05:56:11,896 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(952): ClusterId : f9dc232b-22f9-451c-8836-8ef2290ef3ad 2023-04-06 05:56:11,901 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-06 05:56:11,906 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-06 05:56:11,906 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-06 05:56:11,907 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-06 05:56:11,909 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-06 05:56:11,910 DEBUG [RS:0;jenkins-hbase20:33239] zookeeper.ReadOnlyZKClient(139): Connect 0x30d140d4 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:11,914 DEBUG [RS:0;jenkins-hbase20:33239] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e16a8be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:11,915 DEBUG [RS:0;jenkins-hbase20:33239] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a292ba4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 05:56:11,918 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 05:56:11,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:11,921 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680760601921 2023-04-06 05:56:11,924 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-06 05:56:11,928 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 05:56:11,928 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-06 05:56:11,933 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:11,938 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-06 05:56:11,943 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:33239 2023-04-06 05:56:11,946 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-06 05:56:11,946 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-06 05:56:11,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-06 05:56:11,947 INFO [RS:0;jenkins-hbase20:33239] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-06 05:56:11,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-06 05:56:11,947 INFO [RS:0;jenkins-hbase20:33239] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-06 05:56:11,947 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1023): About to register with Master. 2023-04-06 05:56:11,948 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:11,949 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-06 05:56:11,950 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,35907,1680760569837 with isa=jenkins-hbase20.apache.org/148.251.75.209:33239, startcode=1680760570949 2023-04-06 05:56:11,951 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-06 05:56:11,952 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-06 05:56:11,955 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-06 05:56:11,955 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-06 05:56:11,962 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680760571957,5,FailOnTimeoutGroup] 2023-04-06 05:56:11,962 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680760571962,5,FailOnTimeoutGroup] 2023-04-06 05:56:11,962 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:11,963 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-06 05:56:11,968 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:11,969 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:11,973 DEBUG [RS:0;jenkins-hbase20:33239] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-06 05:56:12,015 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40823, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-06 05:56:12,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,037 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1 2023-04-06 05:56:12,037 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:32913 2023-04-06 05:56:12,037 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-06 05:56:12,042 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 05:56:12,043 DEBUG [RS:0;jenkins-hbase20:33239] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,043 WARN [RS:0;jenkins-hbase20:33239] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-06 05:56:12,043 INFO [RS:0;jenkins-hbase20:33239] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 05:56:12,044 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/WALs/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,044 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,33239,1680760570949] 2023-04-06 05:56:12,054 DEBUG [RS:0;jenkins-hbase20:33239] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,066 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-06 05:56:12,075 INFO [RS:0;jenkins-hbase20:33239] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-06 05:56:12,091 INFO [RS:0;jenkins-hbase20:33239] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-06 05:56:12,094 INFO [RS:0;jenkins-hbase20:33239] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-06 05:56:12,094 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,095 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-06 05:56:12,101 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,101 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,102 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,102 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,102 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,102 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,102 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-06 05:56:12,103 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,103 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,103 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,103 DEBUG [RS:0;jenkins-hbase20:33239] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-06 05:56:12,104 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,104 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,104 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,117 INFO [RS:0;jenkins-hbase20:33239] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-06 05:56:12,119 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,33239,1680760570949-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:12,132 INFO [RS:0;jenkins-hbase20:33239] regionserver.Replication(203): jenkins-hbase20.apache.org,33239,1680760570949 started 2023-04-06 05:56:12,132 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,33239,1680760570949, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:33239, sessionid=0x10084d914bb0001 2023-04-06 05:56:12,132 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-06 05:56:12,132 DEBUG [RS:0;jenkins-hbase20:33239] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,132 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33239,1680760570949' 2023-04-06 05:56:12,132 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:12,133 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,33239,1680760570949' 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-06 05:56:12,134 DEBUG [RS:0;jenkins-hbase20:33239] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-06 05:56:12,135 DEBUG [RS:0;jenkins-hbase20:33239] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-06 05:56:12,135 INFO [RS:0;jenkins-hbase20:33239] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-06 05:56:12,135 INFO [RS:0;jenkins-hbase20:33239] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-06 05:56:12,250 INFO [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33239%2C1680760570949, suffix=, logDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/WALs/jenkins-hbase20.apache.org,33239,1680760570949, archiveDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/oldWALs, maxLogs=32 2023-04-06 05:56:12,267 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK] 2023-04-06 05:56:12,271 INFO [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/WALs/jenkins-hbase20.apache.org,33239,1680760570949/jenkins-hbase20.apache.org%2C33239%2C1680760570949.1680760572253 2023-04-06 05:56:12,271 DEBUG [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK]] 2023-04-06 05:56:12,378 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:12,381 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:12,381 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1 2023-04-06 05:56:12,800 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:12,806 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 05:56:12,810 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/info 2023-04-06 05:56:12,810 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 05:56:12,812 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:12,812 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 05:56:12,815 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/rep_barrier 2023-04-06 05:56:12,815 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 05:56:12,816 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:12,816 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 05:56:12,819 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/table 2023-04-06 05:56:12,819 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 05:56:12,820 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:12,822 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740 2023-04-06 05:56:12,823 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740 2023-04-06 05:56:12,827 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:56:12,829 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 05:56:12,832 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:56:12,833 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73964844, jitterRate=0.10216206312179565}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:56:12,833 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 05:56:12,833 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 05:56:12,833 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 05:56:12,833 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 05:56:12,833 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 05:56:12,834 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 05:56:12,834 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 05:56:12,835 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 05:56:12,840 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-06 05:56:12,840 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-06 05:56:12,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-06 05:56:12,858 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-06 05:56:12,860 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-06 05:56:13,014 DEBUG [jenkins-hbase20:35907] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-06 05:56:13,020 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,33239,1680760570949, state=OPENING 2023-04-06 05:56:13,032 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-06 05:56:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:13,034 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 05:56:13,036 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:13,224 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:13,226 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-06 05:56:13,230 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-06 05:56:13,240 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-06 05:56:13,241 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-06 05:56:13,244 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C33239%2C1680760570949.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/WALs/jenkins-hbase20.apache.org,33239,1680760570949, archiveDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/oldWALs, maxLogs=32 2023-04-06 05:56:13,261 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK] 2023-04-06 05:56:13,266 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/WALs/jenkins-hbase20.apache.org,33239,1680760570949/jenkins-hbase20.apache.org%2C33239%2C1680760570949.meta.1680760573246.meta 2023-04-06 05:56:13,266 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46105,DS-213faee0-9388-4ac4-ab6e-cd500a8d49d3,DISK]] 2023-04-06 05:56:13,266 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:13,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-06 05:56:13,283 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-06 05:56:13,287 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-06 05:56:13,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-06 05:56:13,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:13,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-06 05:56:13,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-06 05:56:13,294 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-06 05:56:13,295 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/info 2023-04-06 05:56:13,295 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/info 2023-04-06 05:56:13,296 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-06 05:56:13,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:13,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-06 05:56:13,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/rep_barrier 2023-04-06 05:56:13,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/rep_barrier 2023-04-06 05:56:13,299 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-06 05:56:13,300 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:13,300 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-06 05:56:13,302 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/table 2023-04-06 05:56:13,302 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/table 2023-04-06 05:56:13,302 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-06 05:56:13,303 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:13,305 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740 2023-04-06 05:56:13,308 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740 2023-04-06 05:56:13,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:56:13,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-06 05:56:13,315 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74843096, jitterRate=0.11524903774261475}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:56:13,315 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-06 05:56:13,323 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680760573216 2023-04-06 05:56:13,338 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-06 05:56:13,338 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-06 05:56:13,339 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,33239,1680760570949, state=OPEN 2023-04-06 05:56:13,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-06 05:56:13,341 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-06 05:56:13,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-06 05:56:13,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,33239,1680760570949 in 305 msec 2023-04-06 05:56:13,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-06 05:56:13,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 498 msec 2023-04-06 05:56:13,355 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5020 sec 2023-04-06 05:56:13,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680760573355, completionTime=-1 2023-04-06 05:56:13,355 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-06 05:56:13,356 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-06 05:56:13,408 DEBUG [hconnection-0x1979d8f0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:13,412 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:13,427 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-06 05:56:13,428 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680760633427 2023-04-06 05:56:13,428 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680760693428 2023-04-06 05:56:13,428 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 72 msec 2023-04-06 05:56:13,451 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35907,1680760569837-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:13,451 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35907,1680760569837-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:13,451 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35907,1680760569837-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:13,452 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:35907, period=300000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:13,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-06 05:56:13,458 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-06 05:56:13,465 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-06 05:56:13,466 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:13,474 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-06 05:56:13,477 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:56:13,480 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:56:13,500 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/hbase/namespace/c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:13,502 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/hbase/namespace/c823e58660018648a7124b90ca124d5e empty. 2023-04-06 05:56:13,503 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/hbase/namespace/c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:13,503 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-06 05:56:13,531 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:13,533 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c823e58660018648a7124b90ca124d5e, NAME => 'hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:56:13,947 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:13,947 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c823e58660018648a7124b90ca124d5e, disabling compactions & flushes 2023-04-06 05:56:13,947 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:13,947 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:13,947 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. after waiting 0 ms 2023-04-06 05:56:13,948 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:13,948 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:13,948 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c823e58660018648a7124b90ca124d5e: 2023-04-06 05:56:13,953 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:56:13,967 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680760573956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760573956"}]},"ts":"1680760573956"} 2023-04-06 05:56:13,990 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:56:13,994 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:56:13,997 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760573994"}]},"ts":"1680760573994"} 2023-04-06 05:56:14,001 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-06 05:56:14,007 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c823e58660018648a7124b90ca124d5e, ASSIGN}] 2023-04-06 05:56:14,011 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c823e58660018648a7124b90ca124d5e, ASSIGN 2023-04-06 05:56:14,013 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c823e58660018648a7124b90ca124d5e, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:56:14,165 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c823e58660018648a7124b90ca124d5e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:14,165 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680760574164"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760574164"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760574164"}]},"ts":"1680760574164"} 2023-04-06 05:56:14,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c823e58660018648a7124b90ca124d5e, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:14,344 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:14,345 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c823e58660018648a7124b90ca124d5e, NAME => 'hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:14,346 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,347 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:14,347 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,347 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,350 INFO [StoreOpener-c823e58660018648a7124b90ca124d5e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,352 DEBUG [StoreOpener-c823e58660018648a7124b90ca124d5e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/info 2023-04-06 05:56:14,352 DEBUG [StoreOpener-c823e58660018648a7124b90ca124d5e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/info 2023-04-06 05:56:14,352 INFO [StoreOpener-c823e58660018648a7124b90ca124d5e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c823e58660018648a7124b90ca124d5e columnFamilyName info 2023-04-06 05:56:14,353 INFO [StoreOpener-c823e58660018648a7124b90ca124d5e-1] regionserver.HStore(310): Store=c823e58660018648a7124b90ca124d5e/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:14,355 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,356 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,360 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c823e58660018648a7124b90ca124d5e 2023-04-06 05:56:14,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:56:14,365 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c823e58660018648a7124b90ca124d5e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74625846, jitterRate=0.11201176047325134}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-06 05:56:14,365 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c823e58660018648a7124b90ca124d5e: 2023-04-06 05:56:14,366 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e., pid=6, masterSystemTime=1680760574333 2023-04-06 05:56:14,370 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:14,370 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:56:14,371 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c823e58660018648a7124b90ca124d5e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:14,372 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680760574371"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760574371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760574371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760574371"}]},"ts":"1680760574371"} 2023-04-06 05:56:14,379 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-06 05:56:14,379 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c823e58660018648a7124b90ca124d5e, server=jenkins-hbase20.apache.org,33239,1680760570949 in 199 msec 2023-04-06 05:56:14,383 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-06 05:56:14,384 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c823e58660018648a7124b90ca124d5e, ASSIGN in 372 msec 2023-04-06 05:56:14,385 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:56:14,386 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760574386"}]},"ts":"1680760574386"} 2023-04-06 05:56:14,389 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-06 05:56:14,392 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:56:14,394 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 925 msec 2023-04-06 05:56:14,479 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-06 05:56:14,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-06 05:56:14,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:56:14,525 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-06 05:56:14,544 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 05:56:14,550 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-06 05:56:14,560 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-06 05:56:14,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-06 05:56:14,578 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-06 05:56:14,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-06 05:56:14,587 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-06 05:56:14,587 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.555sec 2023-04-06 05:56:14,589 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-06 05:56:14,591 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-06 05:56:14,591 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-06 05:56:14,592 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35907,1680760569837-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-06 05:56:14,593 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,35907,1680760569837-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-06 05:56:14,603 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x1545112e to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:14,603 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-06 05:56:14,607 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d85639, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:14,621 DEBUG [hconnection-0x3192c45d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:14,630 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:14,639 INFO [Listener at localhost.localdomain/33939] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:56:14,658 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=186, ProcessCount=169, AvailableMemoryMB=4230 2023-04-06 05:56:14,672 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:56:14,675 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:56:14,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:56:14,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:14,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:14,693 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:56:14,696 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:56:14,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-06 05:56:14,699 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:14,700 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 empty. 2023-04-06 05:56:14,702 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:14,702 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:56:14,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 05:56:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 05:56:15,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 05:56:15,123 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:15,125 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3b3b76ed0a0e2f55ca9d4d617dee4930, NAME => 'TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3b3b76ed0a0e2f55ca9d4d617dee4930, disabling compactions & flushes 2023-04-06 05:56:15,141 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. after waiting 0 ms 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,141 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,141 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:15,147 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:56:15,149 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760575149"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760575149"}]},"ts":"1680760575149"} 2023-04-06 05:56:15,151 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:56:15,153 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:56:15,153 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760575153"}]},"ts":"1680760575153"} 2023-04-06 05:56:15,155 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:56:15,158 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, ASSIGN}] 2023-04-06 05:56:15,159 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, ASSIGN 2023-04-06 05:56:15,161 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:56:15,313 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3b3b76ed0a0e2f55ca9d4d617dee4930, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:15,315 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760575313"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760575313"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760575313"}]},"ts":"1680760575313"} 2023-04-06 05:56:15,321 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:15,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 05:56:15,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3b3b76ed0a0e2f55ca9d4d617dee4930, NAME => 'TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:15,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:15,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,493 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,495 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A 2023-04-06 05:56:15,495 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A 2023-04-06 05:56:15,497 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:15,497 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b3b76ed0a0e2f55ca9d4d617dee4930 columnFamilyName A 2023-04-06 05:56:15,498 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(310): Store=3b3b76ed0a0e2f55ca9d4d617dee4930/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:15,499 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,501 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B 2023-04-06 05:56:15,501 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B 2023-04-06 05:56:15,501 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:15,502 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b3b76ed0a0e2f55ca9d4d617dee4930 columnFamilyName B 2023-04-06 05:56:15,503 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(310): Store=3b3b76ed0a0e2f55ca9d4d617dee4930/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:15,503 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,506 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C 2023-04-06 05:56:15,506 DEBUG [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C 2023-04-06 05:56:15,506 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:15,506 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b3b76ed0a0e2f55ca9d4d617dee4930 columnFamilyName C 2023-04-06 05:56:15,507 INFO [StoreOpener-3b3b76ed0a0e2f55ca9d4d617dee4930-1] regionserver.HStore(310): Store=3b3b76ed0a0e2f55ca9d4d617dee4930/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:15,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:56:15,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:15,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:56:15,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3b3b76ed0a0e2f55ca9d4d617dee4930; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63044130, jitterRate=-0.06056925654411316}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:56:15,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:15,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., pid=11, masterSystemTime=1680760575476 2023-04-06 05:56:15,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:15,530 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3b3b76ed0a0e2f55ca9d4d617dee4930, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:15,531 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760575530"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760575530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760575530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760575530"}]},"ts":"1680760575530"} 2023-04-06 05:56:15,538 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-06 05:56:15,538 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 in 213 msec 2023-04-06 05:56:15,542 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-06 05:56:15,542 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, ASSIGN in 381 msec 2023-04-06 05:56:15,544 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:56:15,544 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760575544"}]},"ts":"1680760575544"} 2023-04-06 05:56:15,547 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:56:15,550 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:56:15,554 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 862 msec 2023-04-06 05:56:15,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-06 05:56:15,830 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-06 05:56:15,838 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x1ad5f945 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,843 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2337cbeb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,845 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,849 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,852 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:56:15,855 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:56:15,862 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x142c8aea to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,866 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e837e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,867 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x136956fd to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,875 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3167991b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,875 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x1f537ba4 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,881 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6be8c20b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,882 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x00e18ba3 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,886 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29be3c5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,887 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x127a87ef to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,892 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbd52be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,892 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x655398b8 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,897 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38ebb9f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,898 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x6a7ef79d to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,923 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13112432, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,924 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x776686ab to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,933 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dcc1cc1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,933 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x385dd3c5 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:15,940 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bcd59bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:15,950 DEBUG [hconnection-0x7d2a217f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,950 DEBUG [hconnection-0x79369e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,953 DEBUG [hconnection-0x1011d054-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,953 DEBUG [hconnection-0x5892536a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,955 DEBUG [hconnection-0x24a6b8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,962 DEBUG [hconnection-0x78372f86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,962 DEBUG [hconnection-0x26957d3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,962 DEBUG [hconnection-0x37403035-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,963 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36384, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,963 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36356, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,962 DEBUG [hconnection-0x719f33c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:15,962 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,965 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36390, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,971 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,974 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36398, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:15,976 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,976 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,985 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:15,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:16,002 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:16,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:16,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:16,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:16,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:16,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:16,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:16,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:16,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:16,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:16,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:16,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:16,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:16,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:16,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:16,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:16,032 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:16,034 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:16,034 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:16,035 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:56:16,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:16,049 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:16,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:16,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760636115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760636117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760636128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760636131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760576065, max timestamp=1680760576128Num uniques 859; 2023-04-06 05:56:16,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:16,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760576065, max timestamp=1680760576128Num uniques 859; 2023-04-06 05:56:16,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760576065, max timestamp=1680760576128Num uniques 859; 2023-04-06 05:56:16,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:56:16,195 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/de96530b4e674950ab9f00bc9e049e13 2023-04-06 05:56:16,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760636289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760636292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760636294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760636296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760636300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760636404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760636405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760636405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760636408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760636409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760636611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760636615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760636615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760636617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760636617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,758 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/5ce01d1f0a404e34be3466247b45e05b 2023-04-06 05:56:16,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760636923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760636921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760636918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:16,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760636925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:16,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760636924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,220 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/bcc2aec84f2142fe9bf3bed3480f28bc 2023-04-06 05:56:17,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/de96530b4e674950ab9f00bc9e049e13 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13 2023-04-06 05:56:17,247 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 05:56:17,256 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/5ce01d1f0a404e34be3466247b45e05b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b 2023-04-06 05:56:17,286 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 05:56:17,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/bcc2aec84f2142fe9bf3bed3480f28bc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc 2023-04-06 05:56:17,317 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc, entries=150, sequenceid=13, filesize=11.6 K 2023-04-06 05:56:17,323 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.37 KB/123255 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1288ms, sequenceid=13, compaction requested=false 2023-04-06 05:56:17,325 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-06 05:56:17,328 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:17,328 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:17,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:17,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:17,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:17,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:17,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:17,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7304d84a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:17,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:17,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,348 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:17,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:17,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:17,349 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,349 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:17,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,354 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:17,354 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:17,371 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:17,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:17,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:17,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:17,372 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:17,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,393 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:17,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:17,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:17,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:17,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:17,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:17,428 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:17,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-06 05:56:17,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:17,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760576279, max timestamp=1680760577462Num uniques 1018; 2023-04-06 05:56:17,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:56:17,492 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:56:17,528 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:17,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:17,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:17,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/ba4debee797e4cc7aeafb7d2546cf24b 2023-04-06 05:56:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:17,581 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:17,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:17,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:17,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:17,587 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,587 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:17,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,590 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:17,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:17,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:17,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:17,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:17,604 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:17,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:17,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:17,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:17,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:17,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:17,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:17,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:17,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:17,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:17,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:17,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:17,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53446a80[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:17,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:17,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:17,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:17,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:17,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:17,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:17,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,627 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:17,626 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760637600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:17,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:17,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760637600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,635 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:17,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:17,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:17,637 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:17,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760637606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760637615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760637613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/4e0d19a6039949e5a35e169936d05956 2023-04-06 05:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:17,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:17,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,675 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:17,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:17,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/25f4b7247df0434fab671ffd690a9091 2023-04-06 05:56:17,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760637738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760637739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760637739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760637744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:17,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760637744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/ba4debee797e4cc7aeafb7d2546cf24b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b 2023-04-06 05:56:17,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:17,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:17,778 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 05:56:17,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:17,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/4e0d19a6039949e5a35e169936d05956 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956 2023-04-06 05:56:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:17,806 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:17,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:17,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:17,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:17,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:17,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:17,821 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,822 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:17,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:17,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 05:56:17,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:17,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:17,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:17,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:17,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:17,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:17,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:17,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:17,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:17,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:17,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:17,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:17,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:17,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:17,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:17,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/25f4b7247df0434fab671ffd690a9091 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091 2023-04-06 05:56:17,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e23c5bb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:17,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:17,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,846 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:17,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:17,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:17,850 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,851 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 05:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:17,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:17,859 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:17,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:17,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:17,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:17,859 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:17,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.48 KB/104935, heapSize ~291.24 KB/298232, currentSize=80.51 KB/82440 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 399ms, sequenceid=51, compaction requested=false 2023-04-06 05:56:17,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:17,865 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:17,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:17,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:17,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:17,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:17,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:17,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:17,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:17,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:17,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:17,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:17,890 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:17,890 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:17,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:17,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 05:56:17,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:17,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:17,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:17,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:17,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:17,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:17,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:17,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:17,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:18,016 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:18,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:18,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:18,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,034 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:18,036 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,036 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760637998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760638009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760638032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:18,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:18,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:18,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760638033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760638036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:18,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:18,056 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:18,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:18,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:18,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:18,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:18,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:18,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:18,084 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-06 05:56:18,086 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,086 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,087 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,087 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,088 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-06 05:56:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2b5d0c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:18,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:18,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,091 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:18,091 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,108 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:18,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:18,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:18,110 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:18,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:18,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:18,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:18,110 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:18,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,118 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:18,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:18,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,141 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:18,141 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760638146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760638147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:18,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760638147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760638151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760638161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,241 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:18,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:18,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:18,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:18,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:18,267 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:18,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:18,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:18,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:18,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:18,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:18,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:18,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,287 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:18,287 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:18,287 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:18,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:18,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:18,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:18,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:18,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:18,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:18,303 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,310 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:18,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:18,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b381e0b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:18,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:18,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:18,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:18,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:18,328 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:18,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:18,335 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:18,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:18,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:18,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:18,335 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:18,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,348 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,354 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760638366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:18,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:18,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:18,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,371 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,371 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:18,378 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:18,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760638371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760638370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760638377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760638379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a695283d4821409aa2ca2b944d2c9375 2023-04-06 05:56:18,478 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:18,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:18,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:18,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:18,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:18,489 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:18,489 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:18,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:18,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,493 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:18,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:18,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:18,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:18,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:18,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:18,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:18,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:18,513 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,513 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:18,513 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:18,513 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,517 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:18,517 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:18,517 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,517 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:18,517 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:18,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/4144c0fa9b8b405db26775cff34a11bd 2023-04-06 05:56:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:18,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b587386[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:18,541 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:18,541 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:18,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:18,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:18,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:18,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:18,551 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:18,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:18,557 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:18,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:18,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:18,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:18,557 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:18,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:18,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:18,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,568 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,590 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:18,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:18,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:18,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:18,592 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,592 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:18,592 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:18,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:18,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:18,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/a5cafb7f18414d5eb52433e3b10a7601 2023-04-06 05:56:18,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a695283d4821409aa2ca2b944d2c9375 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375 2023-04-06 05:56:18,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:56:18,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/4144c0fa9b8b405db26775cff34a11bd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd 2023-04-06 05:56:18,692 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:18,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:18,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760638693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:18,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760638696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760638697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760638699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:56:18,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760638700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/a5cafb7f18414d5eb52433e3b10a7601 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601 2023-04-06 05:56:18,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:18,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:18,719 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:18,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:18,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:18,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,724 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:18,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:18,724 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:18,724 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:18,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:18,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:56:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:18,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:18,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 783ms, sequenceid=68, compaction requested=true 2023-04-06 05:56:18,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:18,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:18,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:18,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:18,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:18,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:18,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:18,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:18,757 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:18,757 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:18,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:18,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:18,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:18,762 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 05:56:18,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:18,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:18,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:18,763 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:18,763 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:18,763 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:18,766 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:18,769 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:18,769 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,769 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.8 K 2023-04-06 05:56:18,771 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:18,771 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5ce01d1f0a404e34be3466247b45e05b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680760576005 2023-04-06 05:56:18,772 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:18,776 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:18,776 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.8 K 2023-04-06 05:56:18,778 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4e0d19a6039949e5a35e169936d05956, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760577447 2023-04-06 05:56:18,779 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting de96530b4e674950ab9f00bc9e049e13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680760576005 2023-04-06 05:56:18,783 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4144c0fa9b8b405db26775cff34a11bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:18,785 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ba4debee797e4cc7aeafb7d2546cf24b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760577447 2023-04-06 05:56:18,789 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a695283d4821409aa2ca2b944d2c9375, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:18,819 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/5c058b104fda40eba0382ad3b79dc2b4 2023-04-06 05:56:18,850 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#11 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:18,850 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#10 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:18,873 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:56:18,941 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/25bf1426ea904e3bad8095c611d3492f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/25bf1426ea904e3bad8095c611d3492f 2023-04-06 05:56:18,988 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/995b3d56438440839cbeebecbe539fc1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/995b3d56438440839cbeebecbe539fc1 2023-04-06 05:56:19,010 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 995b3d56438440839cbeebecbe539fc1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:19,010 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:19,010 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=13, startTime=1680760578756; duration=0sec 2023-04-06 05:56:19,011 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:19,011 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:19,012 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 25bf1426ea904e3bad8095c611d3492f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:19,017 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:19,014 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:19,017 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=13, startTime=1680760578751; duration=0sec 2023-04-06 05:56:19,026 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:19,026 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:19,026 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,026 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.8 K 2023-04-06 05:56:19,031 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bcc2aec84f2142fe9bf3bed3480f28bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680760576005 2023-04-06 05:56:19,031 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 25f4b7247df0434fab671ffd690a9091, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760577447 2023-04-06 05:56:19,034 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a5cafb7f18414d5eb52433e3b10a7601, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:19,068 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#13 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:19,111 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/1c2bdf47476147e8885e408b6ac77e90 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1c2bdf47476147e8885e408b6ac77e90 2023-04-06 05:56:19,131 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 1c2bdf47476147e8885e408b6ac77e90(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:19,131 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:19,132 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=13, startTime=1680760578756; duration=0sec 2023-04-06 05:56:19,132 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:19,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:19,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760639233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760639234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760639237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760639238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760639238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,333 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/b972d80d2c27439da2f53e8644a6ea13 2023-04-06 05:56:19,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760639342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760639342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760639347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760639347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760639348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,399 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/ba28e429dd40463bbe6c370281596cc1 2023-04-06 05:56:19,413 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/5c058b104fda40eba0382ad3b79dc2b4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4 2023-04-06 05:56:19,432 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 05:56:19,440 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/b972d80d2c27439da2f53e8644a6ea13 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13 2023-04-06 05:56:19,470 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 05:56:19,477 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/ba28e429dd40463bbe6c370281596cc1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1 2023-04-06 05:56:19,500 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1, entries=150, sequenceid=83, filesize=11.6 K 2023-04-06 05:56:19,501 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 739ms, sequenceid=83, compaction requested=false 2023-04-06 05:56:19,502 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:19,502 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:19,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:19,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:19,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:19,512 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:19,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:19,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:19,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b66794b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:19,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:19,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:19,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:19,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:19,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:19,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:19,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:19,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:19,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:19,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:19,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:19,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:19,542 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,544 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,549 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:19,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:19,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:19,549 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,552 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:19,553 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:19,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:19,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:19,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:19,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:19,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:19,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:19,556 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:19,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/d35e2dc315eb4165a0447012aaa31af0 2023-04-06 05:56:19,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760639580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760639606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760639637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760639642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760639646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,653 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:19,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/23a00174e4284d8eba65477d99de4244 2023-04-06 05:56:19,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:19,672 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:19,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:19,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:19,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:19,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:19,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:19,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:19,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:19,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:19,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:19,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:19,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:19,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:19,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:19,682 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,682 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:19,682 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:19,683 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:19,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:19,687 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:19,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:19,697 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,697 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f919816[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:19,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:19,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,701 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:19,701 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:19,701 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:19,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:19,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:19,703 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:19,703 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:19,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:19,704 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,706 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:19,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:19,712 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:19,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:19,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:19,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:19,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:19,734 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:19,734 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:19,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:19,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/eeadb7d7e9c6403b845e78092d5b0f39 2023-04-06 05:56:19,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760639746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760639747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760639747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760639747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760639749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/d35e2dc315eb4165a0447012aaa31af0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0 2023-04-06 05:56:19,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0, entries=150, sequenceid=106, filesize=11.6 K 2023-04-06 05:56:19,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/23a00174e4284d8eba65477d99de4244 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244 2023-04-06 05:56:19,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244, entries=150, sequenceid=106, filesize=11.6 K 2023-04-06 05:56:19,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/eeadb7d7e9c6403b845e78092d5b0f39 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39 2023-04-06 05:56:19,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39, entries=150, sequenceid=106, filesize=11.6 K 2023-04-06 05:56:19,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 274ms, sequenceid=106, compaction requested=true 2023-04-06 05:56:19,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:19,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:19,830 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:19,830 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:19,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:19,830 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:19,831 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:19,832 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:19,832 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:19,832 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:19,832 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,832 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,832 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/995b3d56438440839cbeebecbe539fc1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.9 K 2023-04-06 05:56:19,832 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/25bf1426ea904e3bad8095c611d3492f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.9 K 2023-04-06 05:56:19,833 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 995b3d56438440839cbeebecbe539fc1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:19,834 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:19,833 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 25bf1426ea904e3bad8095c611d3492f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:19,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:19,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:19,850 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:19,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:19,857 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b972d80d2c27439da2f53e8644a6ea13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680760577983 2023-04-06 05:56:19,857 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5c058b104fda40eba0382ad3b79dc2b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680760577983 2023-04-06 05:56:19,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:19,863 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 23a00174e4284d8eba65477d99de4244, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:19,863 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d35e2dc315eb4165a0447012aaa31af0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:19,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:19,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,879 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:19,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:19,880 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,880 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:19,880 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:19,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:19,892 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#18 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:19,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:19,906 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:19,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:19,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:19,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:19,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:19,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:19,925 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:56:19,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:19,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:19,926 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,926 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:19,926 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:19,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:19,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:19,987 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 05:56:19,988 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-06 05:56:20,009 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/344e90f4f2924e299f87c684bb28969f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/344e90f4f2924e299f87c684bb28969f 2023-04-06 05:56:20,011 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a430908709b34e7da43f9cf541a4c24b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a430908709b34e7da43f9cf541a4c24b 2023-04-06 05:56:20,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760640016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,030 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into a430908709b34e7da43f9cf541a4c24b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:20,037 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:20,037 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=13, startTime=1680760579830; duration=0sec 2023-04-06 05:56:20,038 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:20,038 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:20,042 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:20,042 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:20,043 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,044 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1c2bdf47476147e8885e408b6ac77e90, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=34.9 K 2023-04-06 05:56:20,045 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1c2bdf47476147e8885e408b6ac77e90, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760577605 2023-04-06 05:56:20,046 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ba28e429dd40463bbe6c370281596cc1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680760577983 2023-04-06 05:56:20,047 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting eeadb7d7e9c6403b845e78092d5b0f39, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:20,071 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/4bd41f28704d4b7b9c2da6947b4d4426 2023-04-06 05:56:20,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,071 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 344e90f4f2924e299f87c684bb28969f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:20,074 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:20,074 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=13, startTime=1680760579830; duration=0sec 2023-04-06 05:56:20,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760640021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,074 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:20,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760640022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760640036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760640038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,082 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:20,165 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/25889ea9f2eb4ef0af9f101db0208b9c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25889ea9f2eb4ef0af9f101db0208b9c 2023-04-06 05:56:20,167 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/9f2eb139a8b04830963a10c05a5def96 2023-04-06 05:56:20,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760640165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760640176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760640182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760640187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760640187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,197 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 25889ea9f2eb4ef0af9f101db0208b9c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:20,197 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:20,197 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=13, startTime=1680760579830; duration=0sec 2023-04-06 05:56:20,197 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:20,252 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/cab9860a64ca406eb708e2c9835ddbd0 2023-04-06 05:56:20,265 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/4bd41f28704d4b7b9c2da6947b4d4426 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426 2023-04-06 05:56:20,285 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426, entries=150, sequenceid=119, filesize=11.6 K 2023-04-06 05:56:20,290 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/9f2eb139a8b04830963a10c05a5def96 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96 2023-04-06 05:56:20,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96, entries=150, sequenceid=119, filesize=11.6 K 2023-04-06 05:56:20,306 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/cab9860a64ca406eb708e2c9835ddbd0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0 2023-04-06 05:56:20,324 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0, entries=150, sequenceid=119, filesize=11.6 K 2023-04-06 05:56:20,340 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 415ms, sequenceid=119, compaction requested=false 2023-04-06 05:56:20,340 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:20,340 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:20,341 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:20,341 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:20,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34220fa9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:20,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:20,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,347 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:20,347 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,347 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:20,348 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:20,360 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:20,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:20,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:20,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:20,360 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:20,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:20,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760580010, max timestamp=1680760580374Num uniques 859; 2023-04-06 05:56:20,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:20,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:20,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:20,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:20,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760640427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760640430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760640430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760640433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760640435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/35a69b1fcfa44135a2e09f87ca49de74 2023-04-06 05:56:20,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:20,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:20,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:20,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:20,502 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:20,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,511 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:20,511 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:20,512 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,512 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,513 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:20,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:20,535 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:20,535 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:20,547 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:20,547 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:20,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,551 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760640541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,551 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:20,552 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:20,552 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:20,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760640542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,554 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760640542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:20,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760640543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee9ee8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:20,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:20,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,558 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:20,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:20,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:20,558 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,558 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:20,560 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:20,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760640557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:20,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:20,580 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:20,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:20,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:20,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:20,580 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:20,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:20,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:20,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,608 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:20,608 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:20,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:20,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:20,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:20,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:20,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:20,715 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:20,715 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:20,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:20,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,719 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:20,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:20,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:20,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:20,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:20,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:20,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:20,730 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:20,730 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:20,730 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:20,730 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:20,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:20,741 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@748a041f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:20,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:20,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,753 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:20,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:20,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:20,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,755 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:20,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760640757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760640759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760640759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760640759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:20,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:20,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:20,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:20,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:20,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760640762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,772 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,782 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:20,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:20,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,784 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:20,784 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:20,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:20,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:20,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:20,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:20,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:20,889 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:20,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:20,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:20,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,893 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,893 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:20,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:20,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:20,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:20,895 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:20,897 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,897 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:20,897 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:20,897 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:20,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:20,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379873b0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:20,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:20,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:20,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:20,921 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:20,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:20,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:20,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:20,921 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:20,931 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,931 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,931 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:20,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:20,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,936 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,940 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:20,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:20,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:20,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:20,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:20,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:20,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:20,942 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:20,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c0063572281a4047904bf37b2516475a 2023-04-06 05:56:21,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/a257f56acdaf42df85b75dedb9141b53 2023-04-06 05:56:21,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/35a69b1fcfa44135a2e09f87ca49de74 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74 2023-04-06 05:56:21,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74, entries=250, sequenceid=145, filesize=16.4 K 2023-04-06 05:56:21,026 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c0063572281a4047904bf37b2516475a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a 2023-04-06 05:56:21,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 05:56:21,043 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:21,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/a257f56acdaf42df85b75dedb9141b53 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53 2023-04-06 05:56:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:21,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:21,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 05:56:21,066 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~301.91 KB/309152, currentSize=73.80 KB/75570 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 669ms, sequenceid=145, compaction requested=true 2023-04-06 05:56:21,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:21,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:21,066 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:21,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:21,066 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:21,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:21,069 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:21,069 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40794 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:21,069 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:21,069 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:21,069 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:21,069 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:21,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:21,069 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a430908709b34e7da43f9cf541a4c24b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=39.8 K 2023-04-06 05:56:21,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:21,069 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:21,069 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/344e90f4f2924e299f87c684bb28969f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=35.2 K 2023-04-06 05:56:21,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:21,070 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a430908709b34e7da43f9cf541a4c24b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:21,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:21,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:21,071 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 344e90f4f2924e299f87c684bb28969f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:21,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:21,077 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9f2eb139a8b04830963a10c05a5def96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680760579576 2023-04-06 05:56:21,077 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4bd41f28704d4b7b9c2da6947b4d4426, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680760579576 2023-04-06 05:56:21,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:21,077 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 35a69b1fcfa44135a2e09f87ca49de74, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580019 2023-04-06 05:56:21,077 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c0063572281a4047904bf37b2516475a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580021 2023-04-06 05:56:21,078 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:21,082 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:21,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:21,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:21,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:21,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:21,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:21,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:21,085 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:21,089 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:21,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:21,094 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:21,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:21,094 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:21,094 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 05:56:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:21,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:21,100 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:21,100 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:21,100 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:21,100 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:21,101 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:21,101 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:21,101 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:21,102 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:21,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760641147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760641145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760641150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760641152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760641152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760641254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760641254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760641255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760641260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760641261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760641459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760641459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760641460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760641463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760641465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,554 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/3dd73a29248f45ccbe6ea268683ce806 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/3dd73a29248f45ccbe6ea268683ce806 2023-04-06 05:56:21,556 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/1f7601cff3a5450c894d982e7957afc8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1f7601cff3a5450c894d982e7957afc8 2023-04-06 05:56:21,562 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 3dd73a29248f45ccbe6ea268683ce806(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:21,563 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:21,563 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=13, startTime=1680760581066; duration=0sec 2023-04-06 05:56:21,564 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:21,564 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:21,565 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 1f7601cff3a5450c894d982e7957afc8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:21,566 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:21,566 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/22b27e436c724ec7859d57a6b0856c38 2023-04-06 05:56:21,566 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=13, startTime=1680760581066; duration=0sec 2023-04-06 05:56:21,566 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:21,571 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:21,571 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:21,571 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:21,571 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25889ea9f2eb4ef0af9f101db0208b9c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=35.2 K 2023-04-06 05:56:21,572 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 25889ea9f2eb4ef0af9f101db0208b9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680760579206 2023-04-06 05:56:21,573 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cab9860a64ca406eb708e2c9835ddbd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680760579576 2023-04-06 05:56:21,574 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a257f56acdaf42df85b75dedb9141b53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580021 2023-04-06 05:56:21,586 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:21,614 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/94c63fb89cd245659226460cdbbbf2d8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94c63fb89cd245659226460cdbbbf2d8 2023-04-06 05:56:21,625 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 94c63fb89cd245659226460cdbbbf2d8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:21,625 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:21,625 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=13, startTime=1680760581067; duration=0sec 2023-04-06 05:56:21,626 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:21,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760641764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760641764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760641768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760641768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:21,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:21,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760641769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,030 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/cc3d9cf2c7d642a48e8da13839b20487 2023-04-06 05:56:22,045 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d6317da407a140e68c4107b95d6515e0 2023-04-06 05:56:22,052 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/22b27e436c724ec7859d57a6b0856c38 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38 2023-04-06 05:56:22,059 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38, entries=150, sequenceid=162, filesize=11.8 K 2023-04-06 05:56:22,064 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/cc3d9cf2c7d642a48e8da13839b20487 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487 2023-04-06 05:56:22,072 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487, entries=150, sequenceid=162, filesize=11.8 K 2023-04-06 05:56:22,075 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d6317da407a140e68c4107b95d6515e0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0 2023-04-06 05:56:22,085 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0, entries=150, sequenceid=162, filesize=11.8 K 2023-04-06 05:56:22,089 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 995ms, sequenceid=162, compaction requested=false 2023-04-06 05:56:22,089 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:22,089 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:22,089 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:22,090 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:22,095 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13636dad[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:22,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:22,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:22,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:22,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:22,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:22,109 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:22,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:22,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:22,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:22,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,125 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:22,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:22,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:22,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:22,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:22,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:22,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:22,135 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:22,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:22,235 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:22,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:22,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:22,241 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:22,241 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:22,242 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:22,242 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:22,243 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,243 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,245 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:22,245 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:22,245 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,245 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:22,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:22,248 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:22,248 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:22,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:22,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:22,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:22,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:22,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:22,249 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:22,251 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:22,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:22,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760642295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760642295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760642297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760642300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760642304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760642407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760642407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760642407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760642408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760642408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760642614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760642614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760642615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760642616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760642616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,697 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/4d2543702dc046dfb5c49cff11a26158 2023-04-06 05:56:22,725 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/f5c4810588cc4de79485fd9b905d88ee 2023-04-06 05:56:22,751 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d7f22f35dc914c64bee3bbd8aeca0be4 2023-04-06 05:56:22,757 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/4d2543702dc046dfb5c49cff11a26158 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158 2023-04-06 05:56:22,764 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 05:56:22,765 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/f5c4810588cc4de79485fd9b905d88ee as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee 2023-04-06 05:56:22,774 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 05:56:22,775 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d7f22f35dc914c64bee3bbd8aeca0be4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4 2023-04-06 05:56:22,782 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4, entries=150, sequenceid=181, filesize=11.8 K 2023-04-06 05:56:22,783 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 534ms, sequenceid=181, compaction requested=true 2023-04-06 05:56:22,784 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:22,784 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:22,784 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:22,787 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:22,787 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,787 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:22,787 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:22,789 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,789 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,789 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,789 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:22,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb2b8f7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:22,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:22,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:22,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:22,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:22,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:22,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:22,805 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:22,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:22,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:22,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:22,805 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:22,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:22,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,823 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:22,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,824 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:22,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:22,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:22,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:22,923 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:22,924 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:22,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:22,946 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:22,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:22,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760642946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760642948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760642950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760642952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760642952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:22,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:22,967 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,967 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,968 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:22,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:22,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:22,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:22,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:22,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:22,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:22,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:22,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:22,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:22,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:22,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:22,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,984 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3ce562[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:22,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:22,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:22,988 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:22,988 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,988 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:22,988 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:22,988 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:22,989 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,989 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:22,989 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:22,989 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,000 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,001 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,028 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,028 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760643054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760643054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760643054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760643059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760643059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,128 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,146 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,157 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,157 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,157 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,158 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,158 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,158 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,158 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,158 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,158 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,164 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bcd1cc6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,173 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,175 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,195 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,196 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,200 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,209 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,210 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760643260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760643261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760643266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760643267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760643267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,310 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,314 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,315 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,315 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,317 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,317 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,318 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,320 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,320 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b0617[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,338 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,343 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,343 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,344 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,344 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,344 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,358 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/afb4574b650f4741b11839d431426b34 2023-04-06 05:56:23,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,463 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,471 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df07389[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,476 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,476 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,476 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,479 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,479 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,479 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,479 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,479 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,485 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,486 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760643563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760643563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760643572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760643572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760643574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,587 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,592 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,597 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,597 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,597 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,597 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,598 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557f88a0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,602 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,605 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,605 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,611 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,611 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,711 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,718 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,720 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,736 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,737 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,737 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,737 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,737 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,737 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,741 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,744 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,744 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b785e5[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,759 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,759 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,778 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,779 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,779 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c17ce34600414500a0e104f6175f3ca1 2023-04-06 05:56:23,879 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:23,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:23,883 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:23,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:23,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:23,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:23,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:23,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:23,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:23,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:23,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:23,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,887 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d97569a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:23,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:23,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:23,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:23,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:23,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:23,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:23,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:23,903 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:23,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:23,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:23,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:23,903 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:23,904 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,904 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,904 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:23,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:23,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:23,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:23,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:23,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:23,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:23,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:23,920 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:23,920 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:23,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:23,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,020 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:24,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:24,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:24,031 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:24,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:24,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:24,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:24,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:24,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:24,036 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,036 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:24,038 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,038 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:24,038 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:24,038 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:24,038 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:24,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b51c9a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:24,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:24,056 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:24,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,057 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,057 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,057 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:24,060 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:24,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:24,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:24,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:24,060 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:24,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:24,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760644066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:24,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760644069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:24,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:24,078 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:24,078 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:24,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:24,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:24,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760644074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760644078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:24,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760644080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,178 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:24,184 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:24,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:24,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:24,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:24,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:24,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:24,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:24,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:24,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:24,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:24,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:24,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:24,198 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,199 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,199 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,199 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:24,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a26bac4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:24,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:24,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:24,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:24,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:24,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:24,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:24,211 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:24,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:24,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:24,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:24,211 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:24,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/e4b460db232f41b6b10910e0e5efda81 2023-04-06 05:56:24,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:24,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:24,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:24,220 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:24,220 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:24,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/afb4574b650f4741b11839d431426b34 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34 2023-04-06 05:56:24,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34, entries=150, sequenceid=198, filesize=11.8 K 2023-04-06 05:56:24,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c17ce34600414500a0e104f6175f3ca1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1 2023-04-06 05:56:24,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1, entries=150, sequenceid=198, filesize=11.8 K 2023-04-06 05:56:24,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/e4b460db232f41b6b10910e0e5efda81 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81 2023-04-06 05:56:24,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81, entries=150, sequenceid=198, filesize=11.8 K 2023-04-06 05:56:24,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1347ms, sequenceid=198, compaction requested=true 2023-04-06 05:56:24,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:24,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:24,269 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:24,269 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:24,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:24,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:24,273 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:24,274 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:24,274 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:24,274 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:24,274 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,274 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,274 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1f7601cff3a5450c894d982e7957afc8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=47.3 K 2023-04-06 05:56:24,274 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/3dd73a29248f45ccbe6ea268683ce806, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=47.3 K 2023-04-06 05:56:24,275 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1f7601cff3a5450c894d982e7957afc8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580021 2023-04-06 05:56:24,275 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3dd73a29248f45ccbe6ea268683ce806, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580021 2023-04-06 05:56:24,277 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting cc3d9cf2c7d642a48e8da13839b20487, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680760580425 2023-04-06 05:56:24,279 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 22b27e436c724ec7859d57a6b0856c38, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680760580425 2023-04-06 05:56:24,279 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f5c4810588cc4de79485fd9b905d88ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680760581116 2023-04-06 05:56:24,283 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4d2543702dc046dfb5c49cff11a26158, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680760581116 2023-04-06 05:56:24,284 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c17ce34600414500a0e104f6175f3ca1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:24,284 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting afb4574b650f4741b11839d431426b34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:24,312 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:24,314 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:24,320 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:24,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:24,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:24,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:24,338 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:24,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:24,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:24,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,340 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,340 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:24,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:24,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:24,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:24,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:24,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:24,356 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:24,356 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:24,357 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:24,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:24,358 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:24,365 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/6d37fd8f160442c39a9b13867a2bf4b5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6d37fd8f160442c39a9b13867a2bf4b5 2023-04-06 05:56:24,378 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 6d37fd8f160442c39a9b13867a2bf4b5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:24,379 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:24,379 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=12, startTime=1680760584269; duration=0sec 2023-04-06 05:56:24,379 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:24,379 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:24,381 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:24,381 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:24,381 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,381 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94c63fb89cd245659226460cdbbbf2d8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=47.3 K 2023-04-06 05:56:24,382 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 94c63fb89cd245659226460cdbbbf2d8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760580021 2023-04-06 05:56:24,383 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d6317da407a140e68c4107b95d6515e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1680760580425 2023-04-06 05:56:24,384 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d7f22f35dc914c64bee3bbd8aeca0be4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680760581116 2023-04-06 05:56:24,384 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e4b460db232f41b6b10910e0e5efda81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:24,410 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/bd2ed1ce2ffb480e80ceb662765870c5 2023-04-06 05:56:24,413 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#42 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:24,444 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/feacdb21f0054314b5ff3abff401c100 2023-04-06 05:56:24,464 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/fc2bf688c5084ad9b4024cfbb180671d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/fc2bf688c5084ad9b4024cfbb180671d 2023-04-06 05:56:24,471 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into fc2bf688c5084ad9b4024cfbb180671d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:24,471 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:24,471 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=12, startTime=1680760584270; duration=0sec 2023-04-06 05:56:24,471 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:24,785 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/5a58db4037e04863ad7e67995b319122 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5a58db4037e04863ad7e67995b319122 2023-04-06 05:56:24,792 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 5a58db4037e04863ad7e67995b319122(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:24,792 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:24,792 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=12, startTime=1680760584269; duration=0sec 2023-04-06 05:56:24,792 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:24,859 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/cdc59de893b34db183a4de742e5ad8af 2023-04-06 05:56:24,869 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/bd2ed1ce2ffb480e80ceb662765870c5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5 2023-04-06 05:56:24,877 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5, entries=150, sequenceid=213, filesize=11.8 K 2023-04-06 05:56:24,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/feacdb21f0054314b5ff3abff401c100 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100 2023-04-06 05:56:24,901 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100, entries=150, sequenceid=213, filesize=11.8 K 2023-04-06 05:56:24,914 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/cdc59de893b34db183a4de742e5ad8af as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af 2023-04-06 05:56:24,941 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af, entries=150, sequenceid=213, filesize=11.8 K 2023-04-06 05:56:24,942 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 585ms, sequenceid=213, compaction requested=false 2023-04-06 05:56:24,942 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:24,942 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:24,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:24,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:24,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:24,947 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b48ec8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:24,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:24,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:24,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:24,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:24,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:24,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:24,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:24,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:24,962 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:24,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:24,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:24,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:24,962 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:24,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:24,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:24,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:24,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:24,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:24,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:24,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:24,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:24,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:24,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:24,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:24,990 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:25,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:25,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:25,109 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:25,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:25,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:25,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,111 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:25,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:25,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:25,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:25,121 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,121 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:25,122 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 05:56:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:25,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:25,133 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:25,133 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:25,133 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:25,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:25,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:25,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:25,176 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a2022a36476c489f9b75c051c9a332c2 2023-04-06 05:56:25,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760585168, max timestamp=1680760585188Num uniques 859; 2023-04-06 05:56:25,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760585168, max timestamp=1680760585187Num uniques 859; 2023-04-06 05:56:25,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:56:25,206 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/107b00a1b8cc49068b17ef3d1d527903 2023-04-06 05:56:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760645212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760645212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760645216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760645218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760645218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,240 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/27d2e59308fb440d8a270959653f82e5 2023-04-06 05:56:25,263 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a2022a36476c489f9b75c051c9a332c2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2 2023-04-06 05:56:25,273 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2, entries=100, sequenceid=226, filesize=9.4 K 2023-04-06 05:56:25,276 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/107b00a1b8cc49068b17ef3d1d527903 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903 2023-04-06 05:56:25,288 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903, entries=100, sequenceid=226, filesize=9.4 K 2023-04-06 05:56:25,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/27d2e59308fb440d8a270959653f82e5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5 2023-04-06 05:56:25,296 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5, entries=100, sequenceid=226, filesize=9.4 K 2023-04-06 05:56:25,298 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.26 KB/141575 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 176ms, sequenceid=226, compaction requested=true 2023-04-06 05:56:25,299 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:25,299 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:25,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:25,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:25,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,307 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c02bc2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:25,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:25,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,315 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:25,315 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:25,316 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:25,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:25,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:25,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:25,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:25,322 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:25,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:25,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:25,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:25,322 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:25,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:25,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=144.97 KB heapSize=422.24 KB 2023-04-06 05:56:25,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:25,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:25,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:25,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:25,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:25,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,336 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,340 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,340 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,341 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,341 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:25,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760645331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:25,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:25,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:25,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760645335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760645334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760645336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760645342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,446 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:25,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:25,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760645444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760645449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:25,455 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:25,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:25,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:25,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:25,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:25,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760645457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760645457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760645457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:25,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:25,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:25,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:25,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:25,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:25,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:25,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:25,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:25,472 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,473 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,473 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c9abc6c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:25,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:25,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:25,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:25,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:25,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,476 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:25,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:25,480 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:25,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:25,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:25,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:25,480 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:25,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:25,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:25,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:25,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:25,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:25,598 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:25,598 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:25,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:25,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:25,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:25,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:25,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:25,611 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:25,612 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:25,612 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,612 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:25,612 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:25,612 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:25,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:25,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:25,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:25,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6355dda1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:25,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:25,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:25,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:25,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:25,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:25,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:25,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:25,652 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:25,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:25,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:25,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:25,652 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:25,655 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,655 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760645654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760645655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,660 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,662 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760645664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,665 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760645665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760645667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:25,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:25,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,682 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:25,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,682 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=69.33 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/1baee932a8f44b19a68f4c2de1ed722e 2023-04-06 05:56:25,769 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/eb71830c0fb847abaec27e3750ba72f9 2023-04-06 05:56:25,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:25,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:25,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:25,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:25,786 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:25,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:25,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:25,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,788 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:25,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:25,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:25,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:25,790 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:25,790 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,790 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:25,790 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:25,790 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:25,790 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,790 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:25,790 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:25,791 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,791 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:25,791 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:25,791 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,791 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:25,791 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:25,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:25,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4733ddcd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:25,793 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:25,793 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,794 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,794 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:25,794 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:25,794 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:25,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:25,796 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:25,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:25,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:25,801 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:25,801 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:25,801 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:25,801 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:25,802 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:25,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,804 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:25,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,808 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,808 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,908 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:25,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:25,916 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:25,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:25,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:25,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,918 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,918 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:25,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:25,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:25,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:25,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:25,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:25,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:25,921 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,921 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,922 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,922 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fb60d2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:25,924 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:25,925 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,926 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:25,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:25,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:25,928 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,928 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:25,930 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:25,934 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:25,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:25,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:25,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:25,934 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:25,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:25,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:25,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:25,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:25,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:25,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:25,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:25,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:25,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:25,954 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:25,955 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:25,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36410 deadline: 1680760645958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36426 deadline: 1680760645960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36430 deadline: 1680760645969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36398 deadline: 1680760645983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36372 deadline: 1680760645984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,055 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:26,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:26,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:26,061 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:26,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:26,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:26,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:26,063 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:26,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:26,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:26,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:26,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:26,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:26,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:26,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:26,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:26,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:26,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:26,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:26,080 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,080 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:26,080 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:26,083 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:26,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@180bd75d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:26,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:26,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,096 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:26,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:26,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:26,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:26,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:26,097 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:26,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:26,100 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:26,100 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:26,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:26,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:26,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,103 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:26,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:26,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:26,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:26,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:26,106 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:26,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,106 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:26,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:26,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:26,106 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:26,196 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5ecb3e49a8df48ce8ade57a762f17fdd 2023-04-06 05:56:26,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/1baee932a8f44b19a68f4c2de1ed722e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e 2023-04-06 05:56:26,207 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:26,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:26,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:26,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:26,214 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:26,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:26,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:26,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e, entries=200, sequenceid=260, filesize=14.2 K 2023-04-06 05:56:26,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:26,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:26,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:26,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/eb71830c0fb847abaec27e3750ba72f9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9 2023-04-06 05:56:26,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:26,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:26,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:26,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:26,224 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:26,224 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,224 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:26,224 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:26,224 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,224 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:26,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:26,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9, entries=150, sequenceid=260, filesize=11.9 K 2023-04-06 05:56:26,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5ecb3e49a8df48ce8ade57a762f17fdd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd 2023-04-06 05:56:26,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,228 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,228 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,228 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:26,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557eb928[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:26,233 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:26,233 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,240 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:26,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:26,241 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,241 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:26,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:26,246 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:26,246 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,246 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,246 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd, entries=150, sequenceid=260, filesize=11.9 K 2023-04-06 05:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~144.97 KB/148445, heapSize ~422.20 KB/432328, currentSize=33.54 KB/34350 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 926ms, sequenceid=260, compaction requested=true 2023-04-06 05:56:26,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:26,250 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:26,250 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:26,250 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:26,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:26,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:26,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:26,250 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:26,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:26,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,252 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48684 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:26,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:26,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,252 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,252 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46294 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:26,253 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:26,253 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,253 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,253 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6d37fd8f160442c39a9b13867a2bf4b5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=45.2 K 2023-04-06 05:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,253 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,253 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5a58db4037e04863ad7e67995b319122, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=47.5 K 2023-04-06 05:56:26,254 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6d37fd8f160442c39a9b13867a2bf4b5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,254 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5a58db4037e04863ad7e67995b319122, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:26,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting feacdb21f0054314b5ff3abff401c100, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680760582948 2023-04-06 05:56:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,255 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting bd2ed1ce2ffb480e80ceb662765870c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680760582948 2023-04-06 05:56:26,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 107b00a1b8cc49068b17ef3d1d527903, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680760585095 2023-04-06 05:56:26,256 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a2022a36476c489f9b75c051c9a332c2, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680760585095 2023-04-06 05:56:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting eb71830c0fb847abaec27e3750ba72f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585213 2023-04-06 05:56:26,257 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1baee932a8f44b19a68f4c2de1ed722e, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585209 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:26,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:26,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:26,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:26,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:26,276 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#51 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:26,284 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#52 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:26,322 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/05243cfe9311477b9c9f85e34c9217f9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/05243cfe9311477b9c9f85e34c9217f9 2023-04-06 05:56:26,331 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 05243cfe9311477b9c9f85e34c9217f9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:26,332 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:26,332 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=12, startTime=1680760586250; duration=0sec 2023-04-06 05:56:26,332 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:26,332 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:26,334 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46294 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:26,334 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:26,334 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,334 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/fc2bf688c5084ad9b4024cfbb180671d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=45.2 K 2023-04-06 05:56:26,335 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting fc2bf688c5084ad9b4024cfbb180671d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680760582289 2023-04-06 05:56:26,335 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cdc59de893b34db183a4de742e5ad8af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680760582948 2023-04-06 05:56:26,336 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 27d2e59308fb440d8a270959653f82e5, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=226, earliestPutTs=1680760585095 2023-04-06 05:56:26,336 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5ecb3e49a8df48ce8ade57a762f17fdd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585213 2023-04-06 05:56:26,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:26,358 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:26,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:26,371 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:26,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:26,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:26,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:26,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:26,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:26,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:26,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:26,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:26,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:26,394 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:26,394 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,398 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:26,399 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:26,399 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:26,418 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/19b005fd69964bccb740492a9258f948 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/19b005fd69964bccb740492a9258f948 2023-04-06 05:56:26,430 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 19b005fd69964bccb740492a9258f948(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:26,430 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:26,431 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=12, startTime=1680760586250; duration=0sec 2023-04-06 05:56:26,431 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:26,438 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/0618177f9eb942e78d18545806642fd5 2023-04-06 05:56:26,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:26,512 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/177a6bbc7b714b1cb914f2f9c2510da6 2023-04-06 05:56:26,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760586498, max timestamp=1680760586548Num uniques 859; 2023-04-06 05:56:26,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:26,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760586498, max timestamp=1680760586548Num uniques 859; 2023-04-06 05:56:26,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:56:26,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760586498, max timestamp=1680760586548Num uniques 859; 2023-04-06 05:56:26,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:26,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760586583, max timestamp=1680760586595Num uniques 1018; 2023-04-06 05:56:26,599 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:56:26,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760586583, max timestamp=1680760586595Num uniques 1018; 2023-04-06 05:56:26,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:56:26,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760586583, max timestamp=1680760586595Num uniques 1009; 2023-04-06 05:56:26,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-06 05:56:26,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760586614, max timestamp=1680760586628Num uniques 1027; 2023-04-06 05:56:26,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760586614, max timestamp=1680760586628Num uniques 1027; 2023-04-06 05:56:26,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 05:56:26,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 05:56:26,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760586614, max timestamp=1680760586628Num uniques 1027; 2023-04-06 05:56:26,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 05:56:26,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760646655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760586651, max timestamp=1680760586656Num uniques 1086; 2023-04-06 05:56:26,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:56:26,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1680760586651, max timestamp=1680760586659Num uniques 1086; 2023-04-06 05:56:26,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=8506, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:56:26,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760586651, max timestamp=1680760586656Num uniques 1086; 2023-04-06 05:56:26,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:56:26,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680760586678, max timestamp=1680760586694Num uniques 1045; 2023-04-06 05:56:26,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-06 05:56:26,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760586686, max timestamp=1680760586694Num uniques 1045; 2023-04-06 05:56:26,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:56:26,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760586686, max timestamp=1680760586694Num uniques 1045; 2023-04-06 05:56:26,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 05:56:26,729 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/5e4c4449adcd4f1e88f14a1a7ad8a80b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5e4c4449adcd4f1e88f14a1a7ad8a80b 2023-04-06 05:56:26,739 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 5e4c4449adcd4f1e88f14a1a7ad8a80b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:26,739 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:26,739 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=12, startTime=1680760586250; duration=0sec 2023-04-06 05:56:26,739 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:26,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760646731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760646733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760646735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760646742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760646769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760646844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760646845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760646845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:26,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760646847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,934 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/94b4ec9f33b8466f8870bcca6c01ead0 2023-04-06 05:56:26,942 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/0618177f9eb942e78d18545806642fd5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5 2023-04-06 05:56:26,950 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5, entries=100, sequenceid=269, filesize=9.5 K 2023-04-06 05:56:26,953 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/177a6bbc7b714b1cb914f2f9c2510da6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6 2023-04-06 05:56:26,963 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6, entries=100, sequenceid=269, filesize=9.5 K 2023-04-06 05:56:26,964 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/94b4ec9f33b8466f8870bcca6c01ead0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0 2023-04-06 05:56:26,971 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0, entries=100, sequenceid=269, filesize=9.5 K 2023-04-06 05:56:26,972 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=149.83 KB/153430 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 574ms, sequenceid=269, compaction requested=false 2023-04-06 05:56:26,972 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:26,973 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:26,973 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:26,973 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:26,973 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,973 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:26,973 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:26,974 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760586729, max timestamp=1680760586976Num uniques 1204; 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760586729, max timestamp=1680760586976Num uniques 1054; 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760586729, max timestamp=1680760586976Num uniques 1104; 2023-04-06 05:56:26,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-06 05:56:26,980 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=26.84 KB heapSize=66.70 KB 2023-04-06 05:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cfacbff[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:26,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:26,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,990 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:26,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:26,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:26,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:26,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:26,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:26,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:26,992 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:26,995 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:26,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:26,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:26,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:26,996 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:26,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:26,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:26,999 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:27,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760587062, max timestamp=1680760587067Num uniques 859; 2023-04-06 05:56:27,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:27,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760587062, max timestamp=1680760587067Num uniques 859; 2023-04-06 05:56:27,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:27,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760587062, max timestamp=1680760587067Num uniques 859; 2023-04-06 05:56:27,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:27,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,110 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,112 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,123 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464dd742[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,126 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,126 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,127 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,128 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:27,149 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,149 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680760587140, max timestamp=1680760587153Num uniques 1068; 2023-04-06 05:56:27,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680760587140, max timestamp=1680760587153Num uniques 1068; 2023-04-06 05:56:27,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-06 05:56:27,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-06 05:56:27,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760587140, max timestamp=1680760587153Num uniques 1068; 2023-04-06 05:56:27,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-06 05:56:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760587171, max timestamp=1680760587180Num uniques 1009; 2023-04-06 05:56:27,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-06 05:56:27,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760587171, max timestamp=1680760587180Num uniques 1127; 2023-04-06 05:56:27,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-06 05:56:27,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760587171, max timestamp=1680760587180Num uniques 1127; 2023-04-06 05:56:27,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-06 05:56:27,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680760587204, max timestamp=1680760587212Num uniques 1036; 2023-04-06 05:56:27,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 05:56:27,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680760587204, max timestamp=1680760587212Num uniques 1036; 2023-04-06 05:56:27,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 05:56:27,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680760587204, max timestamp=1680760587212Num uniques 1036; 2023-04-06 05:56:27,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-06 05:56:27,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680760587223, max timestamp=1680760587240Num uniques 1095; 2023-04-06 05:56:27,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 05:56:27,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680760587223, max timestamp=1680760587240Num uniques 1095; 2023-04-06 05:56:27,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 05:56:27,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680760587223, max timestamp=1680760587240Num uniques 1095; 2023-04-06 05:56:27,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-06 05:56:27,249 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,260 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,260 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,260 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,260 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,262 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760647272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760647274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760647277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,287 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760647278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,288 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760647279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ba41e9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,298 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,298 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,303 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:27,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,307 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,308 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,317 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,327 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,327 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760647383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760647383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760647390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760647390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760647396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,434 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,435 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,454 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202d1f2f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,457 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,457 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,457 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,461 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,461 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/e152827e824e4b0297e43036dddb6370 2023-04-06 05:56:27,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:27,486 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,486 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,586 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760647590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760647590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760647594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760647597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760647598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,602 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,604 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,604 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,604 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,608 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,608 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,608 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,608 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad29de6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,618 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,619 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,619 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,619 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:27,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,626 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,626 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,632 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,632 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,632 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,634 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:27,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,641 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,641 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,741 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,744 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,745 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,745 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,746 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,748 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@528db239[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,750 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:27,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,754 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,760 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,760 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,860 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,865 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:27,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:27,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:27,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:27,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:27,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:27,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:27,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:27,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:27,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:27,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:27,882 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0488[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:27,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:27,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:27,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:27,885 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:27,887 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:27,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:27,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:27,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:27,887 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:27,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:27,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:27,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:27,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:27,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:27,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:27,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:27,891 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:27,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:27,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760647893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760647894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760647898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:27,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760647900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760647902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/a59820d95b1c48b19784012942050344 2023-04-06 05:56:27,991 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:27,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:27,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:27,996 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:27,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:27,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:27,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:27,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:27,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:27,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:27,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:27,999 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:27,999 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,018 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,018 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,018 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,018 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4876b7cd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,039 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,039 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,045 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,057 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,057 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,157 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,186 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,206 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,206 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26253e88[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,221 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,221 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,222 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,222 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,222 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,233 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,234 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,234 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,233 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,234 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,234 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,334 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,350 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,350 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,350 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,350 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/85f93fd901b24bc8a21c07c3e7ab9625 2023-04-06 05:56:28,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,351 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/e152827e824e4b0297e43036dddb6370 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370 2023-04-06 05:56:28,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,367 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,367 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,367 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,367 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370, entries=200, sequenceid=379, filesize=14.3 K 2023-04-06 05:56:28,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/a59820d95b1c48b19784012942050344 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344 2023-04-06 05:56:28,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,374 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,374 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344, entries=150, sequenceid=379, filesize=11.9 K 2023-04-06 05:56:28,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19064c0f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/85f93fd901b24bc8a21c07c3e7ab9625 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625 2023-04-06 05:56:28,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,382 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,384 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625, entries=150, sequenceid=379, filesize=11.9 K 2023-04-06 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.58 KB/88656, currentSize=154.31 KB/158010 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1424ms, sequenceid=379, compaction requested=true 2023-04-06 05:56:28,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:28,405 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,405 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:28,406 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,407 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37084 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:28,407 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34644 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:28,407 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:28,407 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:28,407 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,407 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/05243cfe9311477b9c9f85e34c9217f9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=36.2 K 2023-04-06 05:56:28,407 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,407 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5e4c4449adcd4f1e88f14a1a7ad8a80b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=33.8 K 2023-04-06 05:56:28,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:28,408 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 05243cfe9311477b9c9f85e34c9217f9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585213 2023-04-06 05:56:28,408 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5e4c4449adcd4f1e88f14a1a7ad8a80b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585213 2023-04-06 05:56:28,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:28,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=161.02 KB heapSize=447.23 KB 2023-04-06 05:56:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,409 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0618177f9eb942e78d18545806642fd5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680760585331 2023-04-06 05:56:28,409 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,410 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,410 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting e152827e824e4b0297e43036dddb6370, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:28,410 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 177a6bbc7b714b1cb914f2f9c2510da6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680760585331 2023-04-06 05:56:28,412 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a59820d95b1c48b19784012942050344, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760587275, max timestamp=1680760588409Num uniques 1204; 2023-04-06 05:56:28,412 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-06 05:56:28,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760587275, max timestamp=1680760588409Num uniques 1204; 2023-04-06 05:56:28,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-06 05:56:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:28,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:28,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:28,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:28,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:28,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,426 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,426 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,432 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#61 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:28,433 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:28,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760648448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760648449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760648450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760648450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760648451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,530 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,535 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,535 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,535 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,536 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,536 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,536 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,536 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,539 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,541 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,541 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@491855e3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,552 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760648555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760648556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,561 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,561 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760648560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760648561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760648561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,564 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,573 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,573 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,693 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,694 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,696 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,697 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,697 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ad5bd2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,698 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,700 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,702 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,702 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,710 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,710 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,710 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760648760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760648761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760648763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760648764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760648765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,813 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf3e2b1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,820 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,820 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,821 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,823 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,823 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,824 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,852 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/322f73a252a6404793c5785b5907993c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/322f73a252a6404793c5785b5907993c 2023-04-06 05:56:28,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/9160eb3e60e840abbf8db67d3878ca14 2023-04-06 05:56:28,861 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 322f73a252a6404793c5785b5907993c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:28,861 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:28,861 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=13, startTime=1680760588405; duration=0sec 2023-04-06 05:56:28,861 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:28,861 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:28,865 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34644 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:28,865 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:28,866 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,866 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/19b005fd69964bccb740492a9258f948, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=33.8 K 2023-04-06 05:56:28,867 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 19b005fd69964bccb740492a9258f948, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760585213 2023-04-06 05:56:28,870 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/280774a26b074aec88c5e194a6bbc164 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/280774a26b074aec88c5e194a6bbc164 2023-04-06 05:56:28,870 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 94b4ec9f33b8466f8870bcca6c01ead0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680760585331 2023-04-06 05:56:28,872 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 85f93fd901b24bc8a21c07c3e7ab9625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:28,879 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 280774a26b074aec88c5e194a6bbc164(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:28,879 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:28,879 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=13, startTime=1680760588405; duration=0sec 2023-04-06 05:56:28,879 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:28,882 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#64 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:28,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/110887db2e3940c4bb3d7ec2d6ec208b 2023-04-06 05:56:28,914 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d124150e4ffc4ec7a7b0b0d54138c531 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d124150e4ffc4ec7a7b0b0d54138c531 2023-04-06 05:56:28,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5fb63fa79f414e39a18d9f1a0ee8a55a 2023-04-06 05:56:28,922 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into d124150e4ffc4ec7a7b0b0d54138c531(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:28,922 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:28,922 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=13, startTime=1680760588406; duration=0sec 2023-04-06 05:56:28,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/9160eb3e60e840abbf8db67d3878ca14 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14 2023-04-06 05:56:28,922 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:28,927 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:28,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:28,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14, entries=150, sequenceid=486, filesize=11.9 K 2023-04-06 05:56:28,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:28,931 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:28,931 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,931 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:28,931 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:28,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/110887db2e3940c4bb3d7ec2d6ec208b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b 2023-04-06 05:56:28,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,935 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:28,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:28,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:28,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:28,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:28,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:28,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:28,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:28,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:28,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:28,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:28,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:28,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:28,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:28,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:28,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b, entries=150, sequenceid=486, filesize=11.9 K 2023-04-06 05:56:28,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5fb63fa79f414e39a18d9f1a0ee8a55a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a 2023-04-06 05:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103eb7f4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:28,943 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:28,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,948 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:28,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:28,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:28,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a, entries=150, sequenceid=486, filesize=11.9 K 2023-04-06 05:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~76.04 KB/77860, heapSize ~205.45 KB/210384, currentSize=114.05 KB/116790 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 545ms, sequenceid=486, compaction requested=false 2023-04-06 05:56:28,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:28,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:28,954 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:28,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:28,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:28,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:28,955 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:28,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:28,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:28,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:28,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:28,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:28,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:28,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:28,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:28,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:28,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:28,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:28,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:28,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:28,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:29,068 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:29,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,073 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,084 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,084 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,084 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,084 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760649074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,085 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760649079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760649080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760649085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760649085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,090 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,090 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,091 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,091 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b825f5d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,093 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,094 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,094 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,095 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,099 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,102 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,102 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/c2d36bbfd3804e6995c7d6ab0f1a9e05 2023-04-06 05:56:29,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760649186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760649186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760649186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760649188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760649189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,206 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,211 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,213 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,213 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452ff434[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,224 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,224 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,224 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,224 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,225 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,232 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,232 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,232 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,232 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,232 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,265 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,266 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,368 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,370 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,370 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,370 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,371 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,371 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,371 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,371 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,381 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,381 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,381 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@616d704a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760649388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760649388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760649390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760649391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760649391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,399 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,399 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,407 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,407 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,407 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,415 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,415 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,515 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,520 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,520 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,520 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,521 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,521 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,521 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,521 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,522 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@489fa84[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,525 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,527 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,527 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c8e0273730d543f8ac67d1f3947f5945 2023-04-06 05:56:29,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,637 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,638 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,646 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,646 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,646 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,646 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d7825ff[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,652 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,652 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,653 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,653 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,654 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,654 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,656 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,656 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,657 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,660 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,661 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,665 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,666 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,666 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760649692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760649692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760649693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760649693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:29,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760649700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,768 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,774 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ccfe30[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,781 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,783 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,783 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,783 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,783 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,783 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,785 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,799 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,799 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:29,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,899 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:29,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:29,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:29,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:29,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:29,902 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:29,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:29,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:29,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:29,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:29,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:29,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:29,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:29,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:29,914 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:29,914 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:29,914 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:29,914 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:29,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:29,923 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da937a3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:29,925 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:29,925 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:29,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:29,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:29,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:29,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:29,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:29,942 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:29,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:29,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:29,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:29,942 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:29,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:29,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/1b81dbe095ed4ab4b928f31c852ad909 2023-04-06 05:56:29,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,965 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:29,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/c2d36bbfd3804e6995c7d6ab0f1a9e05 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05 2023-04-06 05:56:29,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:29,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:29,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:29,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:29,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:29,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:29,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:29,980 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:29,980 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:29,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05, entries=200, sequenceid=510, filesize=14.3 K 2023-04-06 05:56:29,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/c8e0273730d543f8ac67d1f3947f5945 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945 2023-04-06 05:56:29,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945, entries=150, sequenceid=510, filesize=11.9 K 2023-04-06 05:56:30,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/1b81dbe095ed4ab4b928f31c852ad909 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909 2023-04-06 05:56:30,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909, entries=150, sequenceid=510, filesize=11.9 K 2023-04-06 05:56:30,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 943ms, sequenceid=510, compaction requested=true 2023-04-06 05:56:30,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:30,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:30,010 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:30,011 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:30,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:30,011 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:30,012 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:30,012 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:30,012 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:30,012 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/280774a26b074aec88c5e194a6bbc164, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=38.8 K 2023-04-06 05:56:30,012 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:30,012 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:30,012 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:30,013 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/322f73a252a6404793c5785b5907993c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=36.4 K 2023-04-06 05:56:30,013 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 280774a26b074aec88c5e194a6bbc164, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:30,013 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 9160eb3e60e840abbf8db67d3878ca14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680760587275 2023-04-06 05:56:30,014 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 322f73a252a6404793c5785b5907993c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:30,014 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c2d36bbfd3804e6995c7d6ab0f1a9e05, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588447 2023-04-06 05:56:30,021 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 110887db2e3940c4bb3d7ec2d6ec208b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680760587275 2023-04-06 05:56:30,022 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c8e0273730d543f8ac67d1f3947f5945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588449 2023-04-06 05:56:30,055 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:30,058 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#70 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:30,080 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:30,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:30,092 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:30,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:30,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:30,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:30,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:30,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:30,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:30,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:30,094 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:30,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:30,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:30,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:30,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:30,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:30,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:30,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:30,108 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:56:30,108 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:30,111 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:30,112 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/f9f0aa54aed948088e118d93df2b54d3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f9f0aa54aed948088e118d93df2b54d3 2023-04-06 05:56:30,121 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into f9f0aa54aed948088e118d93df2b54d3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:30,121 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:30,121 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=13, startTime=1680760590010; duration=0sec 2023-04-06 05:56:30,121 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:30,121 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:56:30,123 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:56:30,123 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:30,123 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:30,123 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d124150e4ffc4ec7a7b0b0d54138c531, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=36.4 K 2023-04-06 05:56:30,124 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d124150e4ffc4ec7a7b0b0d54138c531, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=379, earliestPutTs=1680760586729 2023-04-06 05:56:30,124 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5fb63fa79f414e39a18d9f1a0ee8a55a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680760587275 2023-04-06 05:56:30,124 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1b81dbe095ed4ab4b928f31c852ad909, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588449 2023-04-06 05:56:30,135 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#72 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:30,159 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/2306aff920364d8aaac6c7c0862c5a56 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2306aff920364d8aaac6c7c0862c5a56 2023-04-06 05:56:30,166 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/6ab8a3629d5c4bb9828a458cee6cba1d 2023-04-06 05:56:30,168 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 2306aff920364d8aaac6c7c0862c5a56(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:30,168 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:30,168 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=13, startTime=1680760590010; duration=0sec 2023-04-06 05:56:30,168 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:30,180 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/49efcfdbe3e74983ada67dd7d3e8acf3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/49efcfdbe3e74983ada67dd7d3e8acf3 2023-04-06 05:56:30,187 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 49efcfdbe3e74983ada67dd7d3e8acf3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:30,187 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:30,187 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=13, startTime=1680760590011; duration=0sec 2023-04-06 05:56:30,187 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:30,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:30,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:30,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760590205, max timestamp=1680760590228Num uniques 859; 2023-04-06 05:56:30,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760590205, max timestamp=1680760590228Num uniques 859; 2023-04-06 05:56:30,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:30,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:30,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760590205, max timestamp=1680760590228Num uniques 859; 2023-04-06 05:56:30,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:30,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760650240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760650240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760650243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760650243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760650244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760650345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760650345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760650346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760650347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760650347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760650549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760650549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760650550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760650551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760650551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,590 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/e187a2f1ffb44dba9229144a1fa46d12 2023-04-06 05:56:30,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760650852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760650852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760650853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760650854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:30,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:30,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760650854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,021 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/4145f4212f4844189396f1ac2cb4b743 2023-04-06 05:56:31,026 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/6ab8a3629d5c4bb9828a458cee6cba1d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d 2023-04-06 05:56:31,032 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d, entries=100, sequenceid=521, filesize=9.5 K 2023-04-06 05:56:31,033 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/e187a2f1ffb44dba9229144a1fa46d12 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12 2023-04-06 05:56:31,039 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12, entries=100, sequenceid=521, filesize=9.5 K 2023-04-06 05:56:31,040 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/4145f4212f4844189396f1ac2cb4b743 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743 2023-04-06 05:56:31,047 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743, entries=100, sequenceid=521, filesize=9.5 K 2023-04-06 05:56:31,047 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 941ms, sequenceid=521, compaction requested=false 2023-04-06 05:56:31,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:31,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:31,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:31,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:31,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:31,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:31,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:31,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:31,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763fa748[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:31,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:31,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,051 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:31,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:31,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:31,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:31,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:31,054 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:31,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:31,056 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:31,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:31,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:31,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:31,056 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:31,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:31,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:31,058 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:31,059 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:31,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:31,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:31,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:31,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:31,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:31,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:31,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:31,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:31,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:31,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:31,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:31,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:31,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:31,164 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:31,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:31,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:31,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:31,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:31,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:31,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:31,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:31,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:31,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:31,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:31,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:31,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:31,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:31,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:31,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:31,167 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-06 05:56:31,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:31,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:31,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:31,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:31,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:31,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:31,179 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a0bae47b244745be811f214be86d53ba 2023-04-06 05:56:31,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:31,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:31,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760651361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760651362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760651363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760651364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760651364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760651465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760651465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760651466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760651466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760651466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,612 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/9af068801bb44d9baa2bcd61f45f286e 2023-04-06 05:56:31,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760651666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760651667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760651670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760651670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760651671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760651969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760651971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760651973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760651973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:31,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:31,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760651974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,025 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/f9fc9f80650b4461b5c6c3fdbf53deba 2023-04-06 05:56:32,029 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a0bae47b244745be811f214be86d53ba as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba 2023-04-06 05:56:32,034 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba, entries=150, sequenceid=559, filesize=11.9 K 2023-04-06 05:56:32,035 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/9af068801bb44d9baa2bcd61f45f286e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e 2023-04-06 05:56:32,040 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e, entries=150, sequenceid=559, filesize=11.9 K 2023-04-06 05:56:32,041 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/f9fc9f80650b4461b5c6c3fdbf53deba as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba 2023-04-06 05:56:32,046 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba, entries=150, sequenceid=559, filesize=11.9 K 2023-04-06 05:56:32,047 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 880ms, sequenceid=559, compaction requested=true 2023-04-06 05:56:32,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:32,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:32,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:32,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:32,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:32,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:32,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:32,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:32,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:32,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aa423ad[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:32,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:32,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:32,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:32,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:32,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,053 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:32,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:32,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:32,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:32,053 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:32,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:32,056 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:32,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:32,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:32,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:32,056 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:32,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:32,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,059 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:32,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:32,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:32,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:32,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:32,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:32,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:32,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:32,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:32,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:32,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:32,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:32,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:32,163 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:32,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:32,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:32,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:32,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:32,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:32,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:32,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:32,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:32,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:32,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:32,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:32,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:32,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:32,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:32,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:32,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:32,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:32,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:32,168 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:56:32,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:32,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:32,169 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:32,169 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:32,169 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:32,169 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:32,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:32,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:32,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760652512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760652512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760652512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760652513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760652514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,584 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/583ddba39be246cb84ba614e41ded63e 2023-04-06 05:56:32,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760652616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760652616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760652616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760652617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760652617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760652818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760652818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760652819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760652819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:32,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760652820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:32,996 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/2484e0de5783437394a71a335a954065 2023-04-06 05:56:33,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760653120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760653121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760653122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760653123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760653123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,406 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/3fe28eb9d34144ab88ff2b4a1c76b316 2023-04-06 05:56:33,412 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/583ddba39be246cb84ba614e41ded63e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e 2023-04-06 05:56:33,417 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e, entries=150, sequenceid=571, filesize=11.9 K 2023-04-06 05:56:33,418 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/2484e0de5783437394a71a335a954065 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065 2023-04-06 05:56:33,423 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065, entries=150, sequenceid=571, filesize=11.9 K 2023-04-06 05:56:33,424 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/3fe28eb9d34144ab88ff2b4a1c76b316 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316 2023-04-06 05:56:33,429 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316, entries=150, sequenceid=571, filesize=11.9 K 2023-04-06 05:56:33,430 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1262ms, sequenceid=571, compaction requested=true 2023-04-06 05:56:33,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:33,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:33,431 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,431 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:33,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac977b7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:33,433 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:33,433 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:33,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:33,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:33,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:33,435 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:33,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:33,435 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:33,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:33,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:33,437 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:33,438 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:33,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:33,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:33,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:33,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:33,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,440 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:33,440 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:33,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:33,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,540 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:33,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:33,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:33,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:33,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:33,550 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:33,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:33,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:33,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:33,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:33,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:33,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:33,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:33,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:33,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:33,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:33,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:33,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:33,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:33,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:33,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:33,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:33,566 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:33,567 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:33,567 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:33,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:33,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:33,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760653629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760653629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760653630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760653633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760653634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760653733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760653733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760653734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760653738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760653738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760653936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760653936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760653936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760653941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:33,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760653941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:33,976 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/6c5f7ef8f7fe405a8d5198aee380ae11 2023-04-06 05:56:34,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760654238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760654240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760654240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760654243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760654245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,393 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/b263f40e7a294f15b3786d152947ef61 2023-04-06 05:56:34,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760654743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760654744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760654747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760654747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:34,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760654750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,810 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2 2023-04-06 05:56:34,815 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/6c5f7ef8f7fe405a8d5198aee380ae11 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11 2023-04-06 05:56:34,820 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:56:34,821 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/b263f40e7a294f15b3786d152947ef61 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61 2023-04-06 05:56:34,825 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:56:34,826 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2 2023-04-06 05:56:34,831 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:56:34,831 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1264ms, sequenceid=591, compaction requested=true 2023-04-06 05:56:34,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:34,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:34,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:34,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,833 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:34,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:34,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:34,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e295162[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:34,834 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:34,834 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,835 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:34,835 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,835 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:34,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:34,836 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:34,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:34,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:34,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:34,838 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:34,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:34,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:34,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:34,838 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:34,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:34,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:34,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:34,843 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:34,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:34,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:34,843 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:34,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:34,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:34,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:34,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:34,946 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:34,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:34,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:34,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:34,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:34,947 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:34,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:34,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:34,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:34,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:34,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:34,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:34,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:34,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:34,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:34,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:34,949 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:34,949 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:34,950 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:34,950 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:35,357 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/74040647d1814d3d8d03c73d104aa529 2023-04-06 05:56:35,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:35,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:35,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760655760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760655761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760655761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760655762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760655764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,772 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/6bd27d44ef504b2d80c7ac4ed90d2342 2023-04-06 05:56:35,782 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/033765241b2d4996bd18bd57c6713b70 2023-04-06 05:56:35,788 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/74040647d1814d3d8d03c73d104aa529 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529 2023-04-06 05:56:35,793 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:56:35,794 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/6bd27d44ef504b2d80c7ac4ed90d2342 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342 2023-04-06 05:56:35,800 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:56:35,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/033765241b2d4996bd18bd57c6713b70 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70 2023-04-06 05:56:35,808 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:56:35,809 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 860ms, sequenceid=604, compaction requested=true 2023-04-06 05:56:35,809 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:35,809 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:35,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:35,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:35,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:35,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:35,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:35,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:35,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15a7b008[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:35,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:35,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,813 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:35,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:35,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:35,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:35,817 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:35,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:35,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:35,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:35,817 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:35,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:35,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:35,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:35,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:35,831 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:35,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:35,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:35,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:35,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:35,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:35,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:35,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760655875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760655879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760655880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760655884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760655884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,931 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:35,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:35,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:35,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:35,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:35,935 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,936 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:35,936 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:35,936 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,936 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:35,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:35,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:35,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:35,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:35,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:35,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:35,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:35,942 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:35,942 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. started... 2023-04-06 05:56:35,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. as already flushing 2023-04-06 05:56:35,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:35,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:35,946 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a7ef79d to 127.0.0.1:55165 2023-04-06 05:56:35,946 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:35,947 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x655398b8 to 127.0.0.1:55165 2023-04-06 05:56:35,947 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:35,947 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,948 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x385dd3c5 to 127.0.0.1:55165 2023-04-06 05:56:35,948 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:35,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:35,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:35,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:35,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e85505a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:35,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:35,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:35,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:35,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:35,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:35,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:35,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:35,954 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:35,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:35,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:35,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:35,954 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:35,955 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:35,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:35,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:35,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:35,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,957 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776686ab to 127.0.0.1:55165 2023-04-06 05:56:35,957 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:35,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:35,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:35,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:35,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:35,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:35,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:35,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:35,958 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:35,958 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:35,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760655985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760655985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760655985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:35,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760655988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:35,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760655988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,058 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:36,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:36,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760656190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760656190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760656190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760656192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760656192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a6996f30446e470092fdbbb1eb679adb 2023-04-06 05:56:36,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760656494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760656495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760656494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760656495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760656497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/1a3fbba96b18491889bb6aef02beae72 2023-04-06 05:56:36,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36398 deadline: 1680760656998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:36,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:36,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36426 deadline: 1680760656998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:37,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36410 deadline: 1680760656999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:37,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:37,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36430 deadline: 1680760657000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:37,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36372 deadline: 1680760657002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:37,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/0e908c01d2ca4a10b474e7f912ef6362 2023-04-06 05:56:37,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/a6996f30446e470092fdbbb1eb679adb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb 2023-04-06 05:56:37,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb, entries=200, sequenceid=625, filesize=14.3 K 2023-04-06 05:56:37,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/1a3fbba96b18491889bb6aef02beae72 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72 2023-04-06 05:56:37,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72, entries=150, sequenceid=625, filesize=11.9 K 2023-04-06 05:56:37,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/0e908c01d2ca4a10b474e7f912ef6362 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362 2023-04-06 05:56:37,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362, entries=150, sequenceid=625, filesize=11.9 K 2023-04-06 05:56:37,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1289ms, sequenceid=625, compaction requested=true 2023-04-06 05:56:37,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:37,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:37,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:37,159 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 05:56:37,159 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 05:56:37,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:37,161 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86082 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 05:56:37,161 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83642 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 05:56:37,161 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/A is initiating minor compaction (all files) 2023-04-06 05:56:37,161 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/B is initiating minor compaction (all files) 2023-04-06 05:56:37,161 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/A in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:37,161 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/B in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:37,161 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f9f0aa54aed948088e118d93df2b54d3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=84.1 K 2023-04-06 05:56:37,161 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2306aff920364d8aaac6c7c0862c5a56, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=81.7 K 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f9f0aa54aed948088e118d93df2b54d3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588449 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2306aff920364d8aaac6c7c0862c5a56, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588449 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 6ab8a3629d5c4bb9828a458cee6cba1d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760589074 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e187a2f1ffb44dba9229144a1fa46d12, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760589074 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a0bae47b244745be811f214be86d53ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680760590241 2023-04-06 05:56:37,162 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9af068801bb44d9baa2bcd61f45f286e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680760590241 2023-04-06 05:56:37,163 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 583ddba39be246cb84ba614e41ded63e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680760591359 2023-04-06 05:56:37,163 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2484e0de5783437394a71a335a954065, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680760591359 2023-04-06 05:56:37,163 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 6c5f7ef8f7fe405a8d5198aee380ae11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760592482 2023-04-06 05:56:37,163 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b263f40e7a294f15b3786d152947ef61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760592482 2023-04-06 05:56:37,164 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 74040647d1814d3d8d03c73d104aa529, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760593628 2023-04-06 05:56:37,164 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6bd27d44ef504b2d80c7ac4ed90d2342, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760593628 2023-04-06 05:56:37,164 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1a3fbba96b18491889bb6aef02beae72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680760595748 2023-04-06 05:56:37,164 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a6996f30446e470092fdbbb1eb679adb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680760595748 2023-04-06 05:56:37,178 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#B#compaction#91 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:37,178 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#A#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:37,593 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/54cee94528384defbb3683eecf9970e1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/54cee94528384defbb3683eecf9970e1 2023-04-06 05:56:37,594 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/7141913e0832446798fc89df77e901bb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/7141913e0832446798fc89df77e901bb 2023-04-06 05:56:37,599 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/B of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 7141913e0832446798fc89df77e901bb(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:37,599 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/A of 3b3b76ed0a0e2f55ca9d4d617dee4930 into 54cee94528384defbb3683eecf9970e1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:37,599 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:37,599 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/A, priority=9, startTime=1680760597159; duration=0sec 2023-04-06 05:56:37,599 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:37,599 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/B, priority=9, startTime=1680760597159; duration=0sec 2023-04-06 05:56:37,599 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:37,599 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:37,599 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-06 05:56:37,601 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83642 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-06 05:56:37,601 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 3b3b76ed0a0e2f55ca9d4d617dee4930/C is initiating minor compaction (all files) 2023-04-06 05:56:37,601 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b3b76ed0a0e2f55ca9d4d617dee4930/C in TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:37,601 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/49efcfdbe3e74983ada67dd7d3e8acf3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp, totalSize=81.7 K 2023-04-06 05:56:37,602 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 49efcfdbe3e74983ada67dd7d3e8acf3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680760588449 2023-04-06 05:56:37,602 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4145f4212f4844189396f1ac2cb4b743, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760589074 2023-04-06 05:56:37,602 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f9fc9f80650b4461b5c6c3fdbf53deba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1680760590241 2023-04-06 05:56:37,602 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3fe28eb9d34144ab88ff2b4a1c76b316, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1680760591359 2023-04-06 05:56:37,603 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2c6e40dd18e24e3d8cda7ef51c3d9bc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760592482 2023-04-06 05:56:37,603 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 033765241b2d4996bd18bd57c6713b70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760593628 2023-04-06 05:56:37,603 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0e908c01d2ca4a10b474e7f912ef6362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1680760595748 2023-04-06 05:56:37,615 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b3b76ed0a0e2f55ca9d4d617dee4930#C#compaction#92 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:38,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:56:38,004 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x127a87ef to 127.0.0.1:55165 2023-04-06 05:56:38,004 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:38,004 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x136956fd to 127.0.0.1:55165 2023-04-06 05:56:38,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,004 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:38,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:38,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,006 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f537ba4 to 127.0.0.1:55165 2023-04-06 05:56:38,007 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,008 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00e18ba3 to 127.0.0.1:55165 2023-04-06 05:56:38,008 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,010 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142c8aea to 127.0.0.1:55165 2023-04-06 05:56:38,010 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,010 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:56:38,010 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-04-06 05:56:38,010 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-06 05:56:38,010 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4576 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4727 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2037 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 6103 rows 2023-04-06 05:56:38,011 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2020 2023-04-06 05:56:38,012 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 6056 rows 2023-04-06 05:56:38,012 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:56:38,012 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ad5f945 to 127.0.0.1:55165 2023-04-06 05:56:38,012 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:56:38,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/f5b5c9f38cde4a4cae28fae63c2b231d 2023-04-06 05:56:38,019 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:56:38,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:56:38,025 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/d31a4ffa20d34b50ab48b3d5d81379f7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d31a4ffa20d34b50ab48b3d5d81379f7 2023-04-06 05:56:38,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/ac231323b0004754a9e8be11d17a8aac 2023-04-06 05:56:38,031 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3b3b76ed0a0e2f55ca9d4d617dee4930/C of 3b3b76ed0a0e2f55ca9d4d617dee4930 into d31a4ffa20d34b50ab48b3d5d81379f7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:38,031 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:38,031 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930., storeName=3b3b76ed0a0e2f55ca9d4d617dee4930/C, priority=9, startTime=1680760597159; duration=0sec 2023-04-06 05:56:38,031 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:38,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:38,035 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760598035"}]},"ts":"1680760598035"} 2023-04-06 05:56:38,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:38,037 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:56:38,103 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:56:38,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, UNASSIGN}] 2023-04-06 05:56:38,107 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, UNASSIGN 2023-04-06 05:56:38,108 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3b3b76ed0a0e2f55ca9d4d617dee4930, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:38,108 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760598108"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760598108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760598108"}]},"ts":"1680760598108"} 2023-04-06 05:56:38,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:38,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:38,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:38,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3b3b76ed0a0e2f55ca9d4d617dee4930, disabling compactions & flushes 2023-04-06 05:56:38,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:38,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5db2558e52994e9a820a0108e2c858c1 2023-04-06 05:56:38,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/f5b5c9f38cde4a4cae28fae63c2b231d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f5b5c9f38cde4a4cae28fae63c2b231d 2023-04-06 05:56:38,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f5b5c9f38cde4a4cae28fae63c2b231d, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:56:38,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/ac231323b0004754a9e8be11d17a8aac as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/ac231323b0004754a9e8be11d17a8aac 2023-04-06 05:56:38,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/ac231323b0004754a9e8be11d17a8aac, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:56:38,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/5db2558e52994e9a820a0108e2c858c1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5db2558e52994e9a820a0108e2c858c1 2023-04-06 05:56:38,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5db2558e52994e9a820a0108e2c858c1, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:56:38,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 465ms, sequenceid=640, compaction requested=false 2023-04-06 05:56:38,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:38,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. after waiting 0 ms 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:38,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 3b3b76ed0a0e2f55ca9d4d617dee4930 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=A 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=B 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b3b76ed0a0e2f55ca9d4d617dee4930, store=C 2023-04-06 05:56:38,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:38,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:38,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/df09ca2e35e3435ba4a0bfd746f4a696 2023-04-06 05:56:39,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:39,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/89737087b6aa41788f9257a23240f286 2023-04-06 05:56:39,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=647 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/c39c3af88f7f4261b0091d66972fe858 2023-04-06 05:56:39,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/A/df09ca2e35e3435ba4a0bfd746f4a696 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/df09ca2e35e3435ba4a0bfd746f4a696 2023-04-06 05:56:39,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/df09ca2e35e3435ba4a0bfd746f4a696, entries=150, sequenceid=647, filesize=11.9 K 2023-04-06 05:56:39,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/B/89737087b6aa41788f9257a23240f286 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/89737087b6aa41788f9257a23240f286 2023-04-06 05:56:39,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/89737087b6aa41788f9257a23240f286, entries=150, sequenceid=647, filesize=11.9 K 2023-04-06 05:56:39,745 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/.tmp/C/c39c3af88f7f4261b0091d66972fe858 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/c39c3af88f7f4261b0091d66972fe858 2023-04-06 05:56:39,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/c39c3af88f7f4261b0091d66972fe858, entries=150, sequenceid=647, filesize=11.9 K 2023-04-06 05:56:39,750 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 3b3b76ed0a0e2f55ca9d4d617dee4930 in 1281ms, sequenceid=647, compaction requested=true 2023-04-06 05:56:39,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/25bf1426ea904e3bad8095c611d3492f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a430908709b34e7da43f9cf541a4c24b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/3dd73a29248f45ccbe6ea268683ce806, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5a58db4037e04863ad7e67995b319122, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/05243cfe9311477b9c9f85e34c9217f9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/280774a26b074aec88c5e194a6bbc164, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f9f0aa54aed948088e118d93df2b54d3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb] to archive 2023-04-06 05:56:39,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:56:39,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/de96530b4e674950ab9f00bc9e049e13 2023-04-06 05:56:39,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/ba4debee797e4cc7aeafb7d2546cf24b 2023-04-06 05:56:39,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/25bf1426ea904e3bad8095c611d3492f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/25bf1426ea904e3bad8095c611d3492f 2023-04-06 05:56:39,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a695283d4821409aa2ca2b944d2c9375 2023-04-06 05:56:39,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5c058b104fda40eba0382ad3b79dc2b4 2023-04-06 05:56:39,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a430908709b34e7da43f9cf541a4c24b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a430908709b34e7da43f9cf541a4c24b 2023-04-06 05:56:39,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/d35e2dc315eb4165a0447012aaa31af0 2023-04-06 05:56:39,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4bd41f28704d4b7b9c2da6947b4d4426 2023-04-06 05:56:39,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/35a69b1fcfa44135a2e09f87ca49de74 2023-04-06 05:56:39,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/3dd73a29248f45ccbe6ea268683ce806 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/3dd73a29248f45ccbe6ea268683ce806 2023-04-06 05:56:39,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/22b27e436c724ec7859d57a6b0856c38 2023-04-06 05:56:39,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/4d2543702dc046dfb5c49cff11a26158 2023-04-06 05:56:39,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5a58db4037e04863ad7e67995b319122 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/5a58db4037e04863ad7e67995b319122 2023-04-06 05:56:39,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/afb4574b650f4741b11839d431426b34 2023-04-06 05:56:39,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/bd2ed1ce2ffb480e80ceb662765870c5 2023-04-06 05:56:39,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a2022a36476c489f9b75c051c9a332c2 2023-04-06 05:56:39,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/1baee932a8f44b19a68f4c2de1ed722e 2023-04-06 05:56:39,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/05243cfe9311477b9c9f85e34c9217f9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/05243cfe9311477b9c9f85e34c9217f9 2023-04-06 05:56:39,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/0618177f9eb942e78d18545806642fd5 2023-04-06 05:56:39,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/e152827e824e4b0297e43036dddb6370 2023-04-06 05:56:39,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/280774a26b074aec88c5e194a6bbc164 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/280774a26b074aec88c5e194a6bbc164 2023-04-06 05:56:39,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/9160eb3e60e840abbf8db67d3878ca14 2023-04-06 05:56:39,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/c2d36bbfd3804e6995c7d6ab0f1a9e05 2023-04-06 05:56:39,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f9f0aa54aed948088e118d93df2b54d3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f9f0aa54aed948088e118d93df2b54d3 2023-04-06 05:56:39,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6ab8a3629d5c4bb9828a458cee6cba1d 2023-04-06 05:56:39,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a0bae47b244745be811f214be86d53ba 2023-04-06 05:56:39,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/583ddba39be246cb84ba614e41ded63e 2023-04-06 05:56:39,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/6c5f7ef8f7fe405a8d5198aee380ae11 2023-04-06 05:56:39,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/74040647d1814d3d8d03c73d104aa529 2023-04-06 05:56:39,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/a6996f30446e470092fdbbb1eb679adb 2023-04-06 05:56:39,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/995b3d56438440839cbeebecbe539fc1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/344e90f4f2924e299f87c684bb28969f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1f7601cff3a5450c894d982e7957afc8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6d37fd8f160442c39a9b13867a2bf4b5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5e4c4449adcd4f1e88f14a1a7ad8a80b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/322f73a252a6404793c5785b5907993c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2306aff920364d8aaac6c7c0862c5a56, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72] to archive 2023-04-06 05:56:39,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:56:39,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5ce01d1f0a404e34be3466247b45e05b 2023-04-06 05:56:39,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4e0d19a6039949e5a35e169936d05956 2023-04-06 05:56:39,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/995b3d56438440839cbeebecbe539fc1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/995b3d56438440839cbeebecbe539fc1 2023-04-06 05:56:39,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/4144c0fa9b8b405db26775cff34a11bd 2023-04-06 05:56:39,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b972d80d2c27439da2f53e8644a6ea13 2023-04-06 05:56:39,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/344e90f4f2924e299f87c684bb28969f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/344e90f4f2924e299f87c684bb28969f 2023-04-06 05:56:39,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/23a00174e4284d8eba65477d99de4244 2023-04-06 05:56:39,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9f2eb139a8b04830963a10c05a5def96 2023-04-06 05:56:39,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1f7601cff3a5450c894d982e7957afc8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1f7601cff3a5450c894d982e7957afc8 2023-04-06 05:56:39,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c0063572281a4047904bf37b2516475a 2023-04-06 05:56:39,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/cc3d9cf2c7d642a48e8da13839b20487 2023-04-06 05:56:39,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/f5c4810588cc4de79485fd9b905d88ee 2023-04-06 05:56:39,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6d37fd8f160442c39a9b13867a2bf4b5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6d37fd8f160442c39a9b13867a2bf4b5 2023-04-06 05:56:39,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c17ce34600414500a0e104f6175f3ca1 2023-04-06 05:56:39,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/feacdb21f0054314b5ff3abff401c100 2023-04-06 05:56:39,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/107b00a1b8cc49068b17ef3d1d527903 2023-04-06 05:56:39,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5e4c4449adcd4f1e88f14a1a7ad8a80b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/5e4c4449adcd4f1e88f14a1a7ad8a80b 2023-04-06 05:56:39,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/eb71830c0fb847abaec27e3750ba72f9 2023-04-06 05:56:39,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/177a6bbc7b714b1cb914f2f9c2510da6 2023-04-06 05:56:39,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/322f73a252a6404793c5785b5907993c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/322f73a252a6404793c5785b5907993c 2023-04-06 05:56:39,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/a59820d95b1c48b19784012942050344 2023-04-06 05:56:39,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/110887db2e3940c4bb3d7ec2d6ec208b 2023-04-06 05:56:39,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2306aff920364d8aaac6c7c0862c5a56 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2306aff920364d8aaac6c7c0862c5a56 2023-04-06 05:56:39,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/c8e0273730d543f8ac67d1f3947f5945 2023-04-06 05:56:39,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/e187a2f1ffb44dba9229144a1fa46d12 2023-04-06 05:56:39,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/9af068801bb44d9baa2bcd61f45f286e 2023-04-06 05:56:39,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/2484e0de5783437394a71a335a954065 2023-04-06 05:56:39,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/b263f40e7a294f15b3786d152947ef61 2023-04-06 05:56:39,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/6bd27d44ef504b2d80c7ac4ed90d2342 2023-04-06 05:56:39,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/1a3fbba96b18491889bb6aef02beae72 2023-04-06 05:56:39,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1c2bdf47476147e8885e408b6ac77e90, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25889ea9f2eb4ef0af9f101db0208b9c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94c63fb89cd245659226460cdbbbf2d8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/fc2bf688c5084ad9b4024cfbb180671d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/19b005fd69964bccb740492a9258f948, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d124150e4ffc4ec7a7b0b0d54138c531, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/49efcfdbe3e74983ada67dd7d3e8acf3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362] to archive 2023-04-06 05:56:39,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:56:39,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/bcc2aec84f2142fe9bf3bed3480f28bc 2023-04-06 05:56:39,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25f4b7247df0434fab671ffd690a9091 2023-04-06 05:56:39,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1c2bdf47476147e8885e408b6ac77e90 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1c2bdf47476147e8885e408b6ac77e90 2023-04-06 05:56:39,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a5cafb7f18414d5eb52433e3b10a7601 2023-04-06 05:56:39,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/ba28e429dd40463bbe6c370281596cc1 2023-04-06 05:56:39,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25889ea9f2eb4ef0af9f101db0208b9c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/25889ea9f2eb4ef0af9f101db0208b9c 2023-04-06 05:56:39,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/eeadb7d7e9c6403b845e78092d5b0f39 2023-04-06 05:56:39,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cab9860a64ca406eb708e2c9835ddbd0 2023-04-06 05:56:39,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94c63fb89cd245659226460cdbbbf2d8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94c63fb89cd245659226460cdbbbf2d8 2023-04-06 05:56:39,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/a257f56acdaf42df85b75dedb9141b53 2023-04-06 05:56:39,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d6317da407a140e68c4107b95d6515e0 2023-04-06 05:56:39,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d7f22f35dc914c64bee3bbd8aeca0be4 2023-04-06 05:56:39,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/fc2bf688c5084ad9b4024cfbb180671d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/fc2bf688c5084ad9b4024cfbb180671d 2023-04-06 05:56:39,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/e4b460db232f41b6b10910e0e5efda81 2023-04-06 05:56:39,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/cdc59de893b34db183a4de742e5ad8af 2023-04-06 05:56:39,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/27d2e59308fb440d8a270959653f82e5 2023-04-06 05:56:39,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/19b005fd69964bccb740492a9258f948 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/19b005fd69964bccb740492a9258f948 2023-04-06 05:56:39,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5ecb3e49a8df48ce8ade57a762f17fdd 2023-04-06 05:56:39,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/94b4ec9f33b8466f8870bcca6c01ead0 2023-04-06 05:56:39,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d124150e4ffc4ec7a7b0b0d54138c531 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d124150e4ffc4ec7a7b0b0d54138c531 2023-04-06 05:56:39,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/85f93fd901b24bc8a21c07c3e7ab9625 2023-04-06 05:56:39,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5fb63fa79f414e39a18d9f1a0ee8a55a 2023-04-06 05:56:39,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/49efcfdbe3e74983ada67dd7d3e8acf3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/49efcfdbe3e74983ada67dd7d3e8acf3 2023-04-06 05:56:39,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/1b81dbe095ed4ab4b928f31c852ad909 2023-04-06 05:56:39,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/4145f4212f4844189396f1ac2cb4b743 2023-04-06 05:56:39,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/f9fc9f80650b4461b5c6c3fdbf53deba 2023-04-06 05:56:39,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/3fe28eb9d34144ab88ff2b4a1c76b316 2023-04-06 05:56:39,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/2c6e40dd18e24e3d8cda7ef51c3d9bc2 2023-04-06 05:56:39,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/033765241b2d4996bd18bd57c6713b70 2023-04-06 05:56:39,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/0e908c01d2ca4a10b474e7f912ef6362 2023-04-06 05:56:39,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/recovered.edits/650.seqid, newMaxSeqId=650, maxSeqId=1 2023-04-06 05:56:39,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930. 2023-04-06 05:56:39,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3b3b76ed0a0e2f55ca9d4d617dee4930: 2023-04-06 05:56:39,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:39,908 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3b3b76ed0a0e2f55ca9d4d617dee4930, regionState=CLOSED 2023-04-06 05:56:39,908 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760599908"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760599908"}]},"ts":"1680760599908"} 2023-04-06 05:56:39,910 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-06 05:56:39,910 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 3b3b76ed0a0e2f55ca9d4d617dee4930, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.7990 sec 2023-04-06 05:56:39,911 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-06 05:56:39,911 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b3b76ed0a0e2f55ca9d4d617dee4930, UNASSIGN in 1.8050 sec 2023-04-06 05:56:39,912 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760599912"}]},"ts":"1680760599912"} 2023-04-06 05:56:39,913 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:56:39,914 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:56:39,915 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8880 sec 2023-04-06 05:56:40,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-06 05:56:40,146 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-06 05:56:40,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:56:40,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,162 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,164 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 05:56:40,171 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:40,175 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/recovered.edits] 2023-04-06 05:56:40,178 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/54cee94528384defbb3683eecf9970e1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/54cee94528384defbb3683eecf9970e1 2023-04-06 05:56:40,179 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/df09ca2e35e3435ba4a0bfd746f4a696 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/df09ca2e35e3435ba4a0bfd746f4a696 2023-04-06 05:56:40,181 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f5b5c9f38cde4a4cae28fae63c2b231d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/A/f5b5c9f38cde4a4cae28fae63c2b231d 2023-04-06 05:56:40,183 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/7141913e0832446798fc89df77e901bb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/7141913e0832446798fc89df77e901bb 2023-04-06 05:56:40,185 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/89737087b6aa41788f9257a23240f286 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/89737087b6aa41788f9257a23240f286 2023-04-06 05:56:40,187 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/ac231323b0004754a9e8be11d17a8aac to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/B/ac231323b0004754a9e8be11d17a8aac 2023-04-06 05:56:40,190 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5db2558e52994e9a820a0108e2c858c1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/5db2558e52994e9a820a0108e2c858c1 2023-04-06 05:56:40,191 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/c39c3af88f7f4261b0091d66972fe858 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/c39c3af88f7f4261b0091d66972fe858 2023-04-06 05:56:40,192 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d31a4ffa20d34b50ab48b3d5d81379f7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/C/d31a4ffa20d34b50ab48b3d5d81379f7 2023-04-06 05:56:40,195 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/recovered.edits/650.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930/recovered.edits/650.seqid 2023-04-06 05:56:40,195 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/3b3b76ed0a0e2f55ca9d4d617dee4930 2023-04-06 05:56:40,195 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:56:40,197 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,215 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:56:40,253 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:56:40,254 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,254 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:56:40,254 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760600254"}]},"ts":"9223372036854775807"} 2023-04-06 05:56:40,257 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:56:40,257 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3b3b76ed0a0e2f55ca9d4d617dee4930, NAME => 'TestAcidGuarantees,,1680760574684.3b3b76ed0a0e2f55ca9d4d617dee4930.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:56:40,257 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:56:40,257 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760600257"}]},"ts":"9223372036854775807"} 2023-04-06 05:56:40,259 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:56:40,261 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,261 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 105 msec 2023-04-06 05:56:40,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-06 05:56:40,268 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-06 05:56:40,285 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=236 (was 207) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_584957953_15 at /127.0.0.1:58684 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_584957953_15 at /127.0.0.1:58504 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:33239-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1979d8f0-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_584957953_15 at /127.0.0.1:58562 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1979d8f0-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_584957953_15 at /127.0.0.1:58634 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1979d8f0-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1979d8f0-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=556 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=289 (was 186) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3686 (was 4230) 2023-04-06 05:56:40,298 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=236, OpenFileDescriptor=556, MaxFileDescriptor=60000, SystemLoadAverage=289, ProcessCount=169, AvailableMemoryMB=3686 2023-04-06 05:56:40,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:56:40,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:40,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:40,304 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:56:40,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-06 05:56:40,305 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:56:40,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:40,306 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:40,307 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 empty. 2023-04-06 05:56:40,307 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:40,307 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:56:40,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:40,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:40,720 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:40,725 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fa1e1102963b7802ef3f913061227193, NAME => 'TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:56:40,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:41,138 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:41,139 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fa1e1102963b7802ef3f913061227193, disabling compactions & flushes 2023-04-06 05:56:41,139 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,139 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,139 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. after waiting 0 ms 2023-04-06 05:56:41,139 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,139 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,139 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:41,146 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:56:41,148 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760601148"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760601148"}]},"ts":"1680760601148"} 2023-04-06 05:56:41,151 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:56:41,152 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:56:41,152 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760601152"}]},"ts":"1680760601152"} 2023-04-06 05:56:41,154 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:56:41,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, ASSIGN}] 2023-04-06 05:56:41,158 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, ASSIGN 2023-04-06 05:56:41,159 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:56:41,311 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:41,311 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760601311"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760601311"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760601311"}]},"ts":"1680760601311"} 2023-04-06 05:56:41,313 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:41,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:41,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fa1e1102963b7802ef3f913061227193, NAME => 'TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:41,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:41,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,477 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,480 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A 2023-04-06 05:56:41,480 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A 2023-04-06 05:56:41,480 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:41,481 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName A 2023-04-06 05:56:41,481 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:41,482 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,483 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B 2023-04-06 05:56:41,483 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B 2023-04-06 05:56:41,483 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:41,484 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName B 2023-04-06 05:56:41,484 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:41,484 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,485 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C 2023-04-06 05:56:41,485 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C 2023-04-06 05:56:41,486 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:41,486 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName C 2023-04-06 05:56:41,486 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:41,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:56:41,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:41,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:56:41,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fa1e1102963b7802ef3f913061227193; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73697729, jitterRate=0.09818173944950104}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:56:41,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:41,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., pid=18, masterSystemTime=1680760601465 2023-04-06 05:56:41,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:41,497 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:41,497 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760601497"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760601497"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760601497"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760601497"}]},"ts":"1680760601497"} 2023-04-06 05:56:41,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-06 05:56:41,501 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 in 186 msec 2023-04-06 05:56:41,504 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-06 05:56:41,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, ASSIGN in 345 msec 2023-04-06 05:56:41,505 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:56:41,505 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760601505"}]},"ts":"1680760601505"} 2023-04-06 05:56:41,507 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:56:41,508 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:56:41,510 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2070 sec 2023-04-06 05:56:42,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-06 05:56:42,416 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-06 05:56:42,419 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x4894c837 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:42,429 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ede141e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:42,431 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:42,438 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59630, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:42,441 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:56:42,446 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:56:42,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:56:42,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:56:42,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 05:56:42,874 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:56:42,876 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:56:42,876 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:56:42,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 05:56:42,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, REOPEN/MOVE}] 2023-04-06 05:56:42,896 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, REOPEN/MOVE 2023-04-06 05:56:42,896 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:42,897 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760602896"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760602896"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760602896"}]},"ts":"1680760602896"} 2023-04-06 05:56:42,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:43,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fa1e1102963b7802ef3f913061227193, disabling compactions & flushes 2023-04-06 05:56:43,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. after waiting 0 ms 2023-04-06 05:56:43,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,058 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 05:56:43,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:43,059 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: fa1e1102963b7802ef3f913061227193 to self. 2023-04-06 05:56:43,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,062 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=CLOSED 2023-04-06 05:56:43,063 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760603062"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760603062"}]},"ts":"1680760603062"} 2023-04-06 05:56:43,065 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-06 05:56:43,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 in 166 msec 2023-04-06 05:56:43,066 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=true 2023-04-06 05:56:43,217 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,217 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760603217"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760603217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760603217"}]},"ts":"1680760603217"} 2023-04-06 05:56:43,221 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:56:43,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fa1e1102963b7802ef3f913061227193, NAME => 'TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:56:43,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:56:43,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,383 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,385 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A 2023-04-06 05:56:43,385 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A 2023-04-06 05:56:43,385 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:43,389 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName A 2023-04-06 05:56:43,391 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:43,391 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,392 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B 2023-04-06 05:56:43,392 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B 2023-04-06 05:56:43,393 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:43,393 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName B 2023-04-06 05:56:43,393 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:43,394 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,394 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C 2023-04-06 05:56:43,394 DEBUG [StoreOpener-fa1e1102963b7802ef3f913061227193-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C 2023-04-06 05:56:43,394 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:56:43,395 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fa1e1102963b7802ef3f913061227193 columnFamilyName C 2023-04-06 05:56:43,396 INFO [StoreOpener-fa1e1102963b7802ef3f913061227193-1] regionserver.HStore(310): Store=fa1e1102963b7802ef3f913061227193/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:56:43,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:56:43,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fa1e1102963b7802ef3f913061227193; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66357289, jitterRate=-0.011199340224266052}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:56:43,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:43,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., pid=23, masterSystemTime=1680760603375 2023-04-06 05:56:43,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,404 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,405 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760603404"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760603404"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760603404"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760603404"}]},"ts":"1680760603404"} 2023-04-06 05:56:43,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-06 05:56:43,408 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 in 185 msec 2023-04-06 05:56:43,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-06 05:56:43,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, REOPEN/MOVE in 513 msec 2023-04-06 05:56:43,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-06 05:56:43,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 531 msec 2023-04-06 05:56:43,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 958 msec 2023-04-06 05:56:43,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-06 05:56:43,423 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x0df763e0 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,428 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b3311fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,429 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x633c81ba to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,433 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38da9522, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,433 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x61ad5d1c to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,437 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@661b21c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,438 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x43a14bcf to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,443 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b7748, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,443 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x6577ce5a to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,447 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c411e08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,447 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7605c204 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,452 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52902392, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,452 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x16505096 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,457 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e1da5ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,457 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x03e1990e to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,461 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a1bd04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,462 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x3d6367b2 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:56:43,466 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32c5a0f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:56:43,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:43,474 DEBUG [hconnection-0x5574844a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,475 DEBUG [hconnection-0x4cb6aa63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,476 DEBUG [hconnection-0x235baea6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,476 DEBUG [hconnection-0xcf8ca7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,477 DEBUG [hconnection-0x7ba03580-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,477 DEBUG [hconnection-0x18e23e66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,477 DEBUG [hconnection-0x5657f601-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:43,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:43,482 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:43,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,484 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:43,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:43,484 DEBUG [hconnection-0x5c7924f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,487 DEBUG [hconnection-0x48abffe9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:56:43,487 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59684, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,487 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,488 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,488 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,489 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59692, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,489 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,492 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,498 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,498 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:43,498 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:56:43,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:43,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:43,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:43,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:43,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:43,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:43,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:43,503 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,503 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:43,506 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:43,506 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:43,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:43,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:43,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:43,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:43,515 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:43,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:43,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f50eaf2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:43,516 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:43,516 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,523 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:43,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:43,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:43,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:43,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:43,525 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:43,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:43,527 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:43,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:43,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:43,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:43,528 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:43,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760603533, max timestamp=1680760603536Num uniques 859; 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760603533, max timestamp=1680760603536Num uniques 859; 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680760603525, max timestamp=1680760603536Num uniques 859; 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:56:43,540 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:56:43,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:43,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,541 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,541 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:43,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,541 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,541 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,541 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,566 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406e60ba00138ab41009db82dc55cc232eb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e60ba00138ab41009db82dc55cc232eb 2023-04-06 05:56:43,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760663566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760663573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760663573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760663574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760663574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,641 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:43,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:43,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:43,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:43,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:43,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:43,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:43,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:43,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:43,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:43,653 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:43,653 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,653 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:43,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:43,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:43,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:43,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:43,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:43,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:43,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:43,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,668 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,668 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba61c3b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:43,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:43,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:43,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:43,671 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,672 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:43,672 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,672 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:43,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:43,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:43,672 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:43,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:43,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:43,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:43,672 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:43,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760663676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760663678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760663678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760663678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760663679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,682 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:43,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:43,683 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:43,683 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:43,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,783 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:43,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:43,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:43,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:43,795 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:43,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:43,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:43,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,797 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,797 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,797 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:43,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:43,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:43,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,802 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:43,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:43,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:43,821 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,821 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:43,821 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:43,821 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:43,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:43,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:43,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:43,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,823 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,824 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:43,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5329d4b7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:43,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:43,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:43,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:43,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:43,828 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:43,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:43,831 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:43,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:43,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:43,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:43,831 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:43,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,833 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,833 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,834 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:43,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:43,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:43,840 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:43,840 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:43,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760663878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760663881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760663881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760663882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760663883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,941 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:43,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:43,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:43,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:43,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:43,948 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:43,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:43,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:43,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,950 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:43,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:43,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:43,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:43,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:43,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:43,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:43,958 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,961 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:43,961 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:43,961 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:43,989 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,989 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,990 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,990 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:43,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770a5908[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:43,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:43,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,993 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:43,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:43,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:43,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:43,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:43,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:43,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:43,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:43,995 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:43,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:43,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:43,997 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:43,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:43,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:43,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:43,997 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:43,999 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,999 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,999 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:43,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,006 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/104c94854ad1484aa1132b81153dcac9 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:44,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,014 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:44,014 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,015 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/c453ccdde4a045c4b430a4a17440b369 2023-04-06 05:56:44,114 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:44,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:44,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:44,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:44,120 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:44,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:44,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:44,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,121 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:44,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:44,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:44,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:44,130 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:44,130 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,130 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:44,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:44,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:44,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:44,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:44,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,140 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:44,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f252a61[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:44,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:44,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:44,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:44,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:44,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:44,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:44,146 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:44,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:44,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:44,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:44,146 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:44,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,154 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:44,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:44,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,165 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:44,165 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,165 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760664181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760664184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760664185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760664186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760664186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,265 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:44,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:44,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:44,269 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:44,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:44,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:44,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:44,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:44,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:44,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:44,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:44,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:44,274 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,275 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,275 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:44,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a9a27a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:44,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:44,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:44,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:44,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:44,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:44,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:44,280 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:44,280 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:44,280 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:44,280 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:44,280 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:44,281 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,281 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,281 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,282 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,283 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:44,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:44,284 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,285 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:44,285 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:44,285 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,385 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:44,388 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:44,388 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,388 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:44,388 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:44,389 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,389 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:44,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:44,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:44,392 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,392 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:44,392 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:44,392 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:44,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:44,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:44,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9385034[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:44,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:44,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,395 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:44,395 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:44,395 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:44,395 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:44,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:44,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:44,398 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:44,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:44,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:44,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:44,398 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:44,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:44,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,400 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:44,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:44,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:44,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:44,402 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:44,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,402 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:44,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:44,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/03aec19faa074b13b0a7330e1aab6b61 2023-04-06 05:56:44,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/104c94854ad1484aa1132b81153dcac9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9 2023-04-06 05:56:44,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9, entries=100, sequenceid=15, filesize=18.4 K 2023-04-06 05:56:44,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/c453ccdde4a045c4b430a4a17440b369 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369 2023-04-06 05:56:44,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 05:56:44,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/03aec19faa074b13b0a7330e1aab6b61 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61 2023-04-06 05:56:44,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61, entries=100, sequenceid=15, filesize=9.3 K 2023-04-06 05:56:44,501 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=137.86 KB/141170 for fa1e1102963b7802ef3f913061227193 in 986ms, sequenceid=15, compaction requested=false 2023-04-06 05:56:44,501 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:44,502 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:44,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:44,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:44,509 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:44,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:44,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:44,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:44,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:44,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,511 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,511 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:44,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:44,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:44,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:44,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:44,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:44,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:44,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:44,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:44,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:44,514 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=137.86 KB heapSize=393.31 KB 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:44,514 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:44,570 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304067c9828ed202244b3ba707bfcce50cff7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067c9828ed202244b3ba707bfcce50cff7 2023-04-06 05:56:44,581 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8d63eda6fb024c0884275b48b21b27c7 2023-04-06 05:56:44,598 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/fe55e61316fe4af9bd055daf49f92922 2023-04-06 05:56:44,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:44,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:44,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760664690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760664690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760664693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760664693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760664693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760664794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760664795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760664796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:44,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:44,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760664997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760664999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760665000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,022 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/c3ea54bc32b54aa7a8693fc70a05ab3a 2023-04-06 05:56:45,028 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8d63eda6fb024c0884275b48b21b27c7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7 2023-04-06 05:56:45,033 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7, entries=150, sequenceid=52, filesize=25.1 K 2023-04-06 05:56:45,034 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/fe55e61316fe4af9bd055daf49f92922 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922 2023-04-06 05:56:45,041 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 05:56:45,042 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/c3ea54bc32b54aa7a8693fc70a05ab3a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a 2023-04-06 05:56:45,049 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a, entries=150, sequenceid=52, filesize=11.6 K 2023-04-06 05:56:45,049 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~137.86 KB/141170, heapSize ~393.27 KB/402704, currentSize=46.96 KB/48090 for fa1e1102963b7802ef3f913061227193 in 535ms, sequenceid=52, compaction requested=false 2023-04-06 05:56:45,049 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:45,049 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:45,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:45,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:45,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:45,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,051 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:45,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:45,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cdf9a02[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:45,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,053 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:45,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:45,053 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,053 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:45,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:45,054 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:45,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:45,057 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:45,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:45,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:45,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:45,057 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:45,058 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,058 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,058 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:45,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:45,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,074 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,074 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:45,174 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:45,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:45,177 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:45,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:45,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:45,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:45,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:45,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:45,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:45,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:45,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:45,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:45,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:45,181 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:45,182 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:45,195 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406369879dbc81c43adb4f7107d43f026c7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406369879dbc81c43adb4f7107d43f026c7 2023-04-06 05:56:45,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:45,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:45,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760605316, max timestamp=1680760605323Num uniques 859; 2023-04-06 05:56:45,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:45,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760605316, max timestamp=1680760605323Num uniques 859; 2023-04-06 05:56:45,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760605316, max timestamp=1680760605323Num uniques 859; 2023-04-06 05:56:45,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:45,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:45,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760665385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760665390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760665391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,445 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:56:45,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760665491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760665494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760665494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,601 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/baa1f302d66340d9873fec650c4217c3 2023-04-06 05:56:45,612 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/e752860b42694b539c5e5c211b797a47 2023-04-06 05:56:45,622 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/ab43841f18ce49009e6b6f57c4940887 2023-04-06 05:56:45,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/baa1f302d66340d9873fec650c4217c3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3 2023-04-06 05:56:45,631 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3, entries=150, sequenceid=62, filesize=25.1 K 2023-04-06 05:56:45,632 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/e752860b42694b539c5e5c211b797a47 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47 2023-04-06 05:56:45,641 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 05:56:45,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/ab43841f18ce49009e6b6f57c4940887 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887 2023-04-06 05:56:45,646 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887, entries=150, sequenceid=62, filesize=11.6 K 2023-04-06 05:56:45,647 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fa1e1102963b7802ef3f913061227193 in 466ms, sequenceid=62, compaction requested=true 2023-04-06 05:56:45,647 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:45,647 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:45,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:45,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:45,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5adfdbda[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:45,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:45,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,650 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,650 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:45,650 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:45,650 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:45,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:45,651 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:45,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,651 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,651 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:45,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:45,653 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:45,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:45,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:45,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:45,653 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:45,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:45,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:45,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,658 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:45,659 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:45,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:45,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:45,697 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760665703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760665704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760665704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,711 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040613c569817a3c48a99cd5f890c45bae4a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040613c569817a3c48a99cd5f890c45bae4a 2023-04-06 05:56:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760665707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760665707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,720 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/57176103b82e41ba9a4275ea87f3b7a4 2023-04-06 05:56:45,731 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/14e9c535f47c486a9c669df6d7078250 2023-04-06 05:56:45,759 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:45,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:45,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:45,762 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:45,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:45,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:45,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,763 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:45,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:45,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:45,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:45,766 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:45,766 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,766 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:45,767 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,767 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:45,767 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:45,767 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:45,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:45,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,769 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:45,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc0889[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:45,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:45,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,771 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:45,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:45,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:45,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:45,772 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:45,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:45,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:45,774 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:45,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:45,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:45,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:45,774 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:45,775 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,778 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:45,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,779 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:45,779 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:45,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:45,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760665808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760665808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760665815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:45,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760665815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,879 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:45,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:45,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:45,883 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:45,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:45,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:45,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,883 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:45,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:45,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:45,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:45,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:45,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:45,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:45,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:45,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,887 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,888 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,888 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:45,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484e413a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:45,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:45,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:45,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:45,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:45,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:45,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:45,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:45,894 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:45,894 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:45,894 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:45,894 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:45,894 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:45,895 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,895 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,895 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:45,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:45,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:45,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:45,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:45,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:45,898 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:45,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:45,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:45,899 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:45,899 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:45,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:45,999 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:45,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:46,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:46,002 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:46,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:46,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:46,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,003 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:46,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:46,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:46,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:46,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:46,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:46,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:46,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:46,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:46,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:46,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,006 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,006 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1811cdbb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:46,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:46,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,007 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:46,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:46,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:46,008 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,008 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:46,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:46,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:46,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:46,010 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:46,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:46,012 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:46,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:46,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:46,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:46,012 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:46,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:46,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760666012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:46,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760666012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:46,017 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:46,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:46,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:46,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:46,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:46,018 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:46,018 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,018 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:46,018 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760666017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760666022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,118 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:46,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:46,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:46,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:46,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:46,121 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:46,122 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,122 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:46,122 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:46,122 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,122 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:46,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:46,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:46,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:46,124 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,124 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:46,124 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:46,124 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:46,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:46,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:46,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c34c468[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:46,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,127 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:46,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,127 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:46,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:46,129 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:46,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:46,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:46,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:46,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:46,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:46,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:46,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:46,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:46,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:46,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,134 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,135 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:46,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:46,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,136 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:46,136 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:46,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:46,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/aa1a1f81e4124f9aacdec55a503e74d8 2023-04-06 05:56:46,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/57176103b82e41ba9a4275ea87f3b7a4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4 2023-04-06 05:56:46,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4, entries=200, sequenceid=99, filesize=31.9 K 2023-04-06 05:56:46,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/14e9c535f47c486a9c669df6d7078250 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250 2023-04-06 05:56:46,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 05:56:46,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/aa1a1f81e4124f9aacdec55a503e74d8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8 2023-04-06 05:56:46,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 05:56:46,162 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for fa1e1102963b7802ef3f913061227193 in 465ms, sequenceid=99, compaction requested=true 2023-04-06 05:56:46,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:46,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:46,162 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:46,162 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:46,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:46,162 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/B is initiating minor compaction (all files) 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 102976 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:46,164 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/B in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/A is initiating minor compaction (all files) 2023-04-06 05:56:46,164 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=44.1 K 2023-04-06 05:56:46,164 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/A in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,164 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=100.6 K 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c453ccdde4a045c4b430a4a17440b369, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760603512 2023-04-06 05:56:46,164 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 104c94854ad1484aa1132b81153dcac9, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760603512 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting fe55e61316fe4af9bd055daf49f92922, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680760603556 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 8d63eda6fb024c0884275b48b21b27c7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680760603556 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e752860b42694b539c5e5c211b797a47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680760604688 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting baa1f302d66340d9873fec650c4217c3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680760604688 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 14e9c535f47c486a9c669df6d7078250, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605388 2023-04-06 05:56:46,165 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 57176103b82e41ba9a4275ea87f3b7a4, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605363 2023-04-06 05:56:46,174 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#B#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:46,195 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#A#compaction#111 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:46,236 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:46,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:46,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:46,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:46,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:46,239 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:46,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:46,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:46,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,240 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,240 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:46,240 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,241 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:46,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:46,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:46,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:46,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:46,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:46,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:46,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:46,242 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:46,242 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:46,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:46,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760606328, max timestamp=1680760606334Num uniques 859; 2023-04-06 05:56:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760606328, max timestamp=1680760606334Num uniques 859; 2023-04-06 05:56:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:56:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:56:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760606328, max timestamp=1680760606334Num uniques 859; 2023-04-06 05:56:46,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:56:46,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760666353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760666354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760666355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760666355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760666458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760666458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760666458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760666458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,594 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/d206cff05ffe46ce8de87e2e661734b4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d206cff05ffe46ce8de87e2e661734b4 2023-04-06 05:56:46,599 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fa1e1102963b7802ef3f913061227193/B of fa1e1102963b7802ef3f913061227193 into d206cff05ffe46ce8de87e2e661734b4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:46,599 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:46,599 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/B, priority=12, startTime=1680760606162; duration=0sec 2023-04-06 05:56:46,599 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:46,599 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:56:46,600 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:56:46,600 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/C is initiating minor compaction (all files) 2023-04-06 05:56:46,600 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/C in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:46,601 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=44.1 K 2023-04-06 05:56:46,601 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 03aec19faa074b13b0a7330e1aab6b61, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760603512 2023-04-06 05:56:46,601 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c3ea54bc32b54aa7a8693fc70a05ab3a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680760603556 2023-04-06 05:56:46,602 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ab43841f18ce49009e6b6f57c4940887, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680760604688 2023-04-06 05:56:46,602 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting aa1a1f81e4124f9aacdec55a503e74d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605388 2023-04-06 05:56:46,611 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#C#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:46,632 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/64a25352007140bea676645fc66c6273 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/64a25352007140bea676645fc66c6273 2023-04-06 05:56:46,646 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fa1e1102963b7802ef3f913061227193/C of fa1e1102963b7802ef3f913061227193 into 64a25352007140bea676645fc66c6273(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:46,646 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:46,646 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/C, priority=12, startTime=1680760606162; duration=0sec 2023-04-06 05:56:46,646 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:46,660 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062f24b87fe6964ab6ba7caf1c4a0928f6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f24b87fe6964ab6ba7caf1c4a0928f6 2023-04-06 05:56:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760666661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760666661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760666661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760666662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760666970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760666970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760666970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760666970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,066 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/40ba052bb4984ad18c6df519a2db1a96 2023-04-06 05:56:47,425 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/9c0963c5653041c5800638be28e8cff1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/9c0963c5653041c5800638be28e8cff1 2023-04-06 05:56:47,430 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fa1e1102963b7802ef3f913061227193/A of fa1e1102963b7802ef3f913061227193 into 9c0963c5653041c5800638be28e8cff1(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 05:56:47,430 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:47,430 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/A, priority=12, startTime=1680760606162; duration=1sec 2023-04-06 05:56:47,430 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:47,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760667473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760667474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760667476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760667476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,479 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/bbc925d548664f42b876700dff3091fc 2023-04-06 05:56:47,497 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/2bde5518b99d41d3b5fb57aefba43f48 2023-04-06 05:56:47,502 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/40ba052bb4984ad18c6df519a2db1a96 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96 2023-04-06 05:56:47,506 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96, entries=150, sequenceid=109, filesize=25.1 K 2023-04-06 05:56:47,507 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/bbc925d548664f42b876700dff3091fc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc 2023-04-06 05:56:47,511 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc, entries=150, sequenceid=109, filesize=11.6 K 2023-04-06 05:56:47,512 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/2bde5518b99d41d3b5fb57aefba43f48 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48 2023-04-06 05:56:47,521 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48, entries=150, sequenceid=109, filesize=11.6 K 2023-04-06 05:56:47,522 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for fa1e1102963b7802ef3f913061227193 in 1280ms, sequenceid=109, compaction requested=false 2023-04-06 05:56:47,522 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:47,522 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:47,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:47,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:47,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:47,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:47,523 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:47,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:47,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:47,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:47,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f2f7b27[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:47,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:47,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,525 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:47,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:47,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:47,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:47,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:47,529 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:47,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:47,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:47,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:47,529 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:47,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:47,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:47,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:47,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:47,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:47,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,533 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:47,534 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:47,634 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:47,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:47,636 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:47,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:47,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:47,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:47,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:47,638 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:47,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:47,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:47,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:47,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:47,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:47,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:47,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:47,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:47,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:47,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:47,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:47,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:47,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:47,644 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-06 05:56:47,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:47,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:47,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:47,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:47,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:47,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:47,667 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062b2f68e4d7974ea2948f9593da20d288 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062b2f68e4d7974ea2948f9593da20d288 2023-04-06 05:56:47,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:47,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:47,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760667726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:47,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:47,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760667829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760668032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,075 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/684bb5a2d5d641709091a601f3ed59ef 2023-04-06 05:56:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760668334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760668478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760668480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760668487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760668489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,495 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/5b738f66df2644a0810ee3d242bff237 2023-04-06 05:56:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:48,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760668840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,906 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/a763caaefec84e18bf91d8d09041c89f 2023-04-06 05:56:48,911 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/684bb5a2d5d641709091a601f3ed59ef as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef 2023-04-06 05:56:48,916 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef, entries=150, sequenceid=147, filesize=25.3 K 2023-04-06 05:56:48,917 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/5b738f66df2644a0810ee3d242bff237 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237 2023-04-06 05:56:48,926 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 05:56:48,927 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/a763caaefec84e18bf91d8d09041c89f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f 2023-04-06 05:56:48,937 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f, entries=150, sequenceid=147, filesize=11.8 K 2023-04-06 05:56:48,938 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for fa1e1102963b7802ef3f913061227193 in 1294ms, sequenceid=147, compaction requested=true 2023-04-06 05:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:48,939 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:48,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:48,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6552b669[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:48,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:48,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,955 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:48,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:48,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:48,956 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:48,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:48,957 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:48,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:48,960 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:48,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:48,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:48,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:48,960 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:48,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:48,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:48,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:48,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:48,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:48,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:48,965 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:48,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:48,966 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:48,966 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:48,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:49,066 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:49,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:49,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:49,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:49,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:49,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:49,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:49,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:49,079 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,079 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:49,080 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,129 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304061b3051180b4043b59594fae7d807439c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b3051180b4043b59594fae7d807439c 2023-04-06 05:56:49,535 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/610dc541d6a84c6aa3a06ae1af82ba2d 2023-04-06 05:56:49,553 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/6f28e5c047774ef0919c7603bb2c58cc 2023-04-06 05:56:49,570 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/0a584eedba3d405e982eb04f96b0a866 2023-04-06 05:56:49,575 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/610dc541d6a84c6aa3a06ae1af82ba2d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d 2023-04-06 05:56:49,580 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d, entries=100, sequenceid=157, filesize=18.5 K 2023-04-06 05:56:49,581 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/6f28e5c047774ef0919c7603bb2c58cc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc 2023-04-06 05:56:49,586 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc, entries=100, sequenceid=157, filesize=9.4 K 2023-04-06 05:56:49,587 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/0a584eedba3d405e982eb04f96b0a866 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866 2023-04-06 05:56:49,602 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866, entries=100, sequenceid=157, filesize=9.4 K 2023-04-06 05:56:49,603 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for fa1e1102963b7802ef3f913061227193 in 524ms, sequenceid=157, compaction requested=true 2023-04-06 05:56:49,603 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:49,603 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:49,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:49,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:49,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:49,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,604 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:49,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b8c36d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:49,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:49,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:49,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:49,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:49,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:49,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:49,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:49,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:49,611 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:49,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:49,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:49,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:49,611 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:49,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:49,616 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:49,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,617 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:49,617 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:49,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,717 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:49,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:49,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:49,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:49,721 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:49,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:49,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:49,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,723 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:49,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:49,745 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:49,745 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,745 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:49,745 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:49,745 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:49,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:49,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:49,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:49,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:49,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,747 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:49,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69355ef6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:49,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:49,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,750 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:49,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:49,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:49,751 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:49,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:49,754 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:49,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:49,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:49,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:49,754 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:49,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:49,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,760 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:49,760 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,861 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:49,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:49,865 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:49,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:49,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:49,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:49,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:49,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:49,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:49,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:49,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:49,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:49,870 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,870 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:49,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:49,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:49,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,880 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:49,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d67f6b4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:49,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:49,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,883 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:49,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,884 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:49,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,884 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:49,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:49,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:49,887 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:49,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:49,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:49,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:49,888 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:49,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:49,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:49,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:49,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:49,893 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:49,893 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:49,893 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:49,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:49,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:49,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:49,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:49,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:49,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:49,948 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406f704fe92393c42ddb7b49811f98e2cc5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f704fe92393c42ddb7b49811f98e2cc5 2023-04-06 05:56:49,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760609969, max timestamp=1680760609983Num uniques 859; 2023-04-06 05:56:49,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:56:49,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760609969, max timestamp=1680760609983Num uniques 859; 2023-04-06 05:56:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:56:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760609972, max timestamp=1680760609983Num uniques 859; 2023-04-06 05:56:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:49,993 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:49,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,006 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,007 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,008 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,014 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,014 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,014 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,014 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,015 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,015 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,015 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3df6e08f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,024 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,024 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,025 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,028 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,029 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,030 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,030 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,030 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,031 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,035 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,035 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760670057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,135 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,139 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,141 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,141 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,142 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,142 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,142 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e0eff7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,144 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,144 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,146 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,147 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,148 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:50,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,153 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,153 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760670160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,253 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,256 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,259 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b61d2f3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,261 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,262 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,262 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,264 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,264 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,265 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,268 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,268 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,268 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,268 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,360 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/3b144295cb7a4c4aadc5216644598cdf 2023-04-06 05:56:50,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760670366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,369 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,374 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/bcc9d8ea2cc14f779495441de645f653 2023-04-06 05:56:50,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,379 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,379 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,379 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,379 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,389 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39c14f7d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,391 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,391 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,392 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,392 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,393 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,393 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,396 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,396 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/10c7e92cf95c47f0b321c255c21bb1bd 2023-04-06 05:56:50,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/3b144295cb7a4c4aadc5216644598cdf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf 2023-04-06 05:56:50,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf, entries=300, sequenceid=169, filesize=45.8 K 2023-04-06 05:56:50,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/bcc9d8ea2cc14f779495441de645f653 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653 2023-04-06 05:56:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,416 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,416 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 05:56:50,419 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/10c7e92cf95c47f0b321c255c21bb1bd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd 2023-04-06 05:56:50,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd, entries=150, sequenceid=169, filesize=11.8 K 2023-04-06 05:56:50,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for fa1e1102963b7802ef3f913061227193 in 502ms, sequenceid=169, compaction requested=true 2023-04-06 05:56:50,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:50,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:50,423 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:50,423 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:50,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:50,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:50,424 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143294 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:50,424 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/A is initiating minor compaction (all files) 2023-04-06 05:56:50,425 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/A in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,425 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/9c0963c5653041c5800638be28e8cff1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=139.9 K 2023-04-06 05:56:50,425 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:50,425 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/B is initiating minor compaction (all files) 2023-04-06 05:56:50,425 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 9c0963c5653041c5800638be28e8cff1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605388 2023-04-06 05:56:50,425 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/B in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,425 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d206cff05ffe46ce8de87e2e661734b4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=56.3 K 2023-04-06 05:56:50,425 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 40ba052bb4984ad18c6df519a2db1a96, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760605702 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d206cff05ffe46ce8de87e2e661734b4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605388 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 684bb5a2d5d641709091a601f3ed59ef, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680760606343 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bbc925d548664f42b876700dff3091fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760605702 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 610dc541d6a84c6aa3a06ae1af82ba2d, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680760607723 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5b738f66df2644a0810ee3d242bff237, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680760606343 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3b144295cb7a4c4aadc5216644598cdf, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609887 2023-04-06 05:56:50,426 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6f28e5c047774ef0919c7603bb2c58cc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680760607723 2023-04-06 05:56:50,427 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bcc9d8ea2cc14f779495441de645f653, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609894 2023-04-06 05:56:50,435 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#B#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:50,445 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#A#compaction#127 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:50,466 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/7ffb816805d842f6948e374cacc8f00b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/7ffb816805d842f6948e374cacc8f00b 2023-04-06 05:56:50,473 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/B of fa1e1102963b7802ef3f913061227193 into 7ffb816805d842f6948e374cacc8f00b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:50,473 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:50,473 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/B, priority=11, startTime=1680760610423; duration=0sec 2023-04-06 05:56:50,473 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:50,473 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:50,475 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57624 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:50,475 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/C is initiating minor compaction (all files) 2023-04-06 05:56:50,475 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/C in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,475 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/64a25352007140bea676645fc66c6273, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=56.3 K 2023-04-06 05:56:50,475 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 64a25352007140bea676645fc66c6273, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760605388 2023-04-06 05:56:50,476 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2bde5518b99d41d3b5fb57aefba43f48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760605702 2023-04-06 05:56:50,477 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a763caaefec84e18bf91d8d09041c89f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680760606343 2023-04-06 05:56:50,477 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 0a584eedba3d405e982eb04f96b0a866, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680760607723 2023-04-06 05:56:50,477 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 10c7e92cf95c47f0b321c255c21bb1bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609894 2023-04-06 05:56:50,484 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:50,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-04-06 05:56:50,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:50,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:50,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:50,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:50,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:50,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:50,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760670498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760670499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760670504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760670505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,508 DEBUG [Thread-797] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:50,511 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406083e282ad38e4b9bb9501229c56bb2a7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406083e282ad38e4b9bb9501229c56bb2a7 2023-04-06 05:56:50,518 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,521 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,522 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,523 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,525 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b93252[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,529 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,531 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,531 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,534 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,535 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,535 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,535 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,535 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,535 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,536 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,536 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760670605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760670605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760670607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,636 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,639 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,643 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca9fc85[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,653 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,653 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760670673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,753 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799a57a9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,763 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,767 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:50,769 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,769 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760670809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760670809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760670810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,869 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:50,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:50,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:50,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:50,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:50,872 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:50,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:50,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:50,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:50,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:50,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:50,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:50,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:50,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:50,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:50,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:50,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a934540[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:50,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:50,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:50,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:50,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:50,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:50,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:50,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:50,895 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:50,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:50,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:50,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:50,895 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:50,901 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/4b40509f0ef6457489836d22160a982e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/4b40509f0ef6457489836d22160a982e 2023-04-06 05:56:50,906 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/d480fdc9acf645e6b6e37fc40eb30a0b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/d480fdc9acf645e6b6e37fc40eb30a0b 2023-04-06 05:56:50,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:50,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:50,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:50,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,916 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:50,916 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:50,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:50,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:50,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:50,916 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:50,916 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:50,916 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:50,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:50,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/f552d81fb50d4989a5ff2d81fba2a91a 2023-04-06 05:56:50,922 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/C of fa1e1102963b7802ef3f913061227193 into 4b40509f0ef6457489836d22160a982e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:50,922 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:50,922 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/C, priority=11, startTime=1680760610423; duration=0sec 2023-04-06 05:56:50,923 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:50,923 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/A of fa1e1102963b7802ef3f913061227193 into d480fdc9acf645e6b6e37fc40eb30a0b(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:50,923 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:50,923 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/A, priority=11, startTime=1680760610423; duration=0sec 2023-04-06 05:56:50,923 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:50,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/a599e584cab34f9c92777c5ae973a517 2023-04-06 05:56:50,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/b6cd965409e14b6fbd17e376345e0cd8 2023-04-06 05:56:50,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/f552d81fb50d4989a5ff2d81fba2a91a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a 2023-04-06 05:56:50,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a, entries=200, sequenceid=207, filesize=32.1 K 2023-04-06 05:56:50,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/a599e584cab34f9c92777c5ae973a517 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517 2023-04-06 05:56:50,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517, entries=150, sequenceid=207, filesize=11.8 K 2023-04-06 05:56:50,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/b6cd965409e14b6fbd17e376345e0cd8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8 2023-04-06 05:56:50,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8, entries=150, sequenceid=207, filesize=11.8 K 2023-04-06 05:56:50,985 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~405.25 KB/414976, currentSize=40.25 KB/41220 for fa1e1102963b7802ef3f913061227193 in 498ms, sequenceid=207, compaction requested=false 2023-04-06 05:56:50,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:51,016 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:51,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:51,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:51,019 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:51,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:51,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:51,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:51,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:51,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:51,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:51,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:51,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:51,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:51,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:51,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:51,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:51,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:51,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:51,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:51,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:51,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:51,022 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:51,022 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:51,022 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:51,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:51,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:51,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:51,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1680760611115, max timestamp=1680760611122Num uniques 859; 2023-04-06 05:56:51,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-06 05:56:51,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1680760611115, max timestamp=1680760611122Num uniques 859; 2023-04-06 05:56:51,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1680760611115, max timestamp=1680760611122Num uniques 859; 2023-04-06 05:56:51,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-06 05:56:51,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-06 05:56:51,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760671137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760671138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760671138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760671176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760671240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760671244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760671244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760671444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760671447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760671447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,451 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069f544f9fcb5e4d8bb941c864f06fddf4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069f544f9fcb5e4d8bb941c864f06fddf4 2023-04-06 05:56:51,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760671747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760671749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:51,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760671751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:51,856 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/ed5b69e86c104a6e922945107a41aabf 2023-04-06 05:56:52,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760672187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760672253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760672253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:52,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760672254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,265 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/911dd897f9874472b97a887f7d8dfe9b 2023-04-06 05:56:52,280 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/5d78ae6f393b4cd8b83b365fe48ef8cd 2023-04-06 05:56:52,284 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/ed5b69e86c104a6e922945107a41aabf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf 2023-04-06 05:56:52,288 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf, entries=150, sequenceid=218, filesize=25.3 K 2023-04-06 05:56:52,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/911dd897f9874472b97a887f7d8dfe9b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b 2023-04-06 05:56:52,294 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b, entries=150, sequenceid=218, filesize=11.8 K 2023-04-06 05:56:52,295 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/5d78ae6f393b4cd8b83b365fe48ef8cd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd 2023-04-06 05:56:52,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd, entries=150, sequenceid=218, filesize=11.8 K 2023-04-06 05:56:52,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=161.02 KB/164880 for fa1e1102963b7802ef3f913061227193 in 1277ms, sequenceid=218, compaction requested=true 2023-04-06 05:56:52,300 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:52,300 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:52,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:52,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:52,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:52,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:52,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:52,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3042d5e2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:52,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:52,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,304 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:52,304 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:52,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:52,304 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:52,304 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:52,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:52,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:52,307 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:52,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:52,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:52,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:52,307 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:52,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:52,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:52,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:52,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:52,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,311 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,311 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:52,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:52,411 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:52,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:52,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:52,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:52,414 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:52,414 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:52,414 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:52,414 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:52,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,419 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:52,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:52,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:52,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:52,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:52,421 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:52,421 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:52,421 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:52,421 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:52,421 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:52,421 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:52,421 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:52,421 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=161.02 KB heapSize=422.48 KB 2023-04-06 05:56:52,421 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:52,421 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:52,421 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:52,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:52,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:52,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:52,445 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062dc42631c72a49908b1ad70495e32d1f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062dc42631c72a49908b1ad70495e32d1f 2023-04-06 05:56:52,863 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/319fa6422dbe49f0a186d132508d163f 2023-04-06 05:56:52,883 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/5f337ca647494f0195490c8c40f690d9 2023-04-06 05:56:52,901 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/04741216314f4f9380be55eacced84a9 2023-04-06 05:56:52,905 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/319fa6422dbe49f0a186d132508d163f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f 2023-04-06 05:56:52,909 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f, entries=150, sequenceid=250, filesize=25.3 K 2023-04-06 05:56:52,910 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/5f337ca647494f0195490c8c40f690d9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9 2023-04-06 05:56:52,915 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9, entries=150, sequenceid=250, filesize=11.8 K 2023-04-06 05:56:52,916 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/04741216314f4f9380be55eacced84a9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9 2023-04-06 05:56:52,935 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9, entries=150, sequenceid=250, filesize=11.8 K 2023-04-06 05:56:52,936 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~422.44 KB/432576, currentSize=0 B/0 for fa1e1102963b7802ef3f913061227193 in 514ms, sequenceid=250, compaction requested=true 2023-04-06 05:56:52,936 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:52,936 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:52,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:52,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:52,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:52,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:52,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,938 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,938 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:52,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43fc4abb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:52,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:52,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:52,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:52,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:52,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:52,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:52,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:52,964 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:52,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:52,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:52,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:52,965 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:52,966 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,966 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:52,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,966 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:52,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:52,966 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:52,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:52,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:52,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:52,969 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:52,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:52,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:52,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:52,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:52,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:52,970 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:52,970 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:52,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:52,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:52,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,070 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,074 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,077 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,077 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,077 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:53,077 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,078 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68706a72[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,082 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,083 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,083 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,083 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,086 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,086 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,086 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,086 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,086 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,094 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,094 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,094 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,094 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,194 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,198 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,199 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,200 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,200 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,201 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,201 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,201 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:53,201 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38999b79[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,205 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,205 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,205 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,209 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,209 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,209 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,209 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,209 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,210 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,210 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,210 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,215 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,215 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:53,309 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:53,310 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:53,315 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,320 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,325 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,334 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,334 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,334 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:53,334 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e52b44d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,344 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,347 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,347 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,347 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,347 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,347 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760673342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760613335, max timestamp=1680760613347Num uniques 859; 2023-04-06 05:56:53,349 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:53,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760613335, max timestamp=1680760613347Num uniques 859; 2023-04-06 05:56:53,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:53,350 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,350 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,350 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760613335, max timestamp=1680760613347Num uniques 859; 2023-04-06 05:56:53,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:56:53,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,354 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b0afb5e4518e47eaa1c4dd81960a7a7a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b0afb5e4518e47eaa1c4dd81960a7a7a 2023-04-06 05:56:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,354 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,354 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,354 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:53,369 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/deb912afaf1149d2bcea31e9e0ad227b 2023-04-06 05:56:53,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760673391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760673391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/a0828f1079e24b2f967ff8b467ee58b1 2023-04-06 05:56:53,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760673450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,455 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,459 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,462 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:53,470 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fda8cae[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,477 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,478 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,484 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,484 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,491 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,491 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,491 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,493 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,497 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,498 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,498 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,498 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760673495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760673496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,598 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,601 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,602 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,604 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,604 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,604 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:53,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@541252dd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,611 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,611 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,615 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,615 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:53,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760673655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760673701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760673701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,715 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,718 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,719 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,721 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,721 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,721 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:53,721 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:53,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:53,722 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:53,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a45975[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:53,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:53,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,725 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:53,726 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,726 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:53,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:53,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:53,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:53,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:53,729 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:53,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:53,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:53,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:53,729 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:53,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:53,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:53,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:53,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:53,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,733 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:53,734 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:53,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:53,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:53,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/6514349fb82443c19fd68c54978affe1 2023-04-06 05:56:53,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/deb912afaf1149d2bcea31e9e0ad227b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b 2023-04-06 05:56:53,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b, entries=400, sequenceid=263, filesize=59.8 K 2023-04-06 05:56:53,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/a0828f1079e24b2f967ff8b467ee58b1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1 2023-04-06 05:56:53,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 05:56:53,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/6514349fb82443c19fd68c54978affe1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1 2023-04-06 05:56:53,834 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:53,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:53,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 05:56:53,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:53,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa1e1102963b7802ef3f913061227193 in 527ms, sequenceid=263, compaction requested=true 2023-04-06 05:56:53,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:53,836 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:53,836 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 05:56:53,836 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:53,836 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:53,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:53,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:53,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:53,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:53,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:53,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:53,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:53,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:53,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:53,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:53,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:53,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:53,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,840 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:53,840 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:53,840 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:53,845 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 172086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:53,845 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/A is initiating minor compaction (all files) 2023-04-06 05:56:53,845 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/A in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,845 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/d480fdc9acf645e6b6e37fc40eb30a0b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=168.1 K 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60638 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/B is initiating minor compaction (all files) 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d480fdc9acf645e6b6e37fc40eb30a0b, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609894 2023-04-06 05:56:53,846 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/B in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:53,846 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/7ffb816805d842f6948e374cacc8f00b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=59.2 K 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f552d81fb50d4989a5ff2d81fba2a91a, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680760610047 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7ffb816805d842f6948e374cacc8f00b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609894 2023-04-06 05:56:53,846 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ed5b69e86c104a6e922945107a41aabf, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680760610494 2023-04-06 05:56:53,847 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a599e584cab34f9c92777c5ae973a517, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680760610050 2023-04-06 05:56:53,847 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 319fa6422dbe49f0a186d132508d163f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680760611123 2023-04-06 05:56:53,848 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 911dd897f9874472b97a887f7d8dfe9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680760610494 2023-04-06 05:56:53,848 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting deb912afaf1149d2bcea31e9e0ad227b, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613280 2023-04-06 05:56:53,848 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5f337ca647494f0195490c8c40f690d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680760611123 2023-04-06 05:56:53,848 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a0828f1079e24b2f967ff8b467ee58b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613306 2023-04-06 05:56:53,862 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040661ad3c243f4d449dad9843b868520df8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661ad3c243f4d449dad9843b868520df8 2023-04-06 05:56:53,872 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#B#compaction#143 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:53,873 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#A#compaction#142 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:53,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:53,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760673986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760674004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760674005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760674090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760674189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,192 DEBUG [Thread-791] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:54,280 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8ba38c3cb7fd47619273f1f7e1aa77c6 2023-04-06 05:56:54,290 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/27d1234541724a249c673d1359754fcf 2023-04-06 05:56:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760674292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,296 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/6cbcb999735d4096ac2abb1189806290 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6cbcb999735d4096ac2abb1189806290 2023-04-06 05:56:54,302 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/B of fa1e1102963b7802ef3f913061227193 into 6cbcb999735d4096ac2abb1189806290(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:54,302 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:54,302 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/B, priority=11, startTime=1680760613836; duration=0sec 2023-04-06 05:56:54,302 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:54,302 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:54,306 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60638 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:54,306 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/C is initiating minor compaction (all files) 2023-04-06 05:56:54,306 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/C in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:54,306 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/4b40509f0ef6457489836d22160a982e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=59.2 K 2023-04-06 05:56:54,306 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4b40509f0ef6457489836d22160a982e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680760609894 2023-04-06 05:56:54,307 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b6cd965409e14b6fbd17e376345e0cd8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1680760610050 2023-04-06 05:56:54,307 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5d78ae6f393b4cd8b83b365fe48ef8cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680760610494 2023-04-06 05:56:54,308 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 04741216314f4f9380be55eacced84a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680760611123 2023-04-06 05:56:54,309 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6514349fb82443c19fd68c54978affe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613306 2023-04-06 05:56:54,317 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#C#compaction#146 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:54,327 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/e70c063d114e42af85305930beb84bb6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/e70c063d114e42af85305930beb84bb6 2023-04-06 05:56:54,335 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/C of fa1e1102963b7802ef3f913061227193 into e70c063d114e42af85305930beb84bb6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:54,335 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:54,335 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/C, priority=11, startTime=1680760613836; duration=0sec 2023-04-06 05:56:54,335 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760674511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760674512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59648 deadline: 1680760674519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,523 DEBUG [Thread-797] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:54,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59664 deadline: 1680760674595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,703 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/b92cbd2e601d46c18e011a458330196b 2023-04-06 05:56:54,707 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8ba38c3cb7fd47619273f1f7e1aa77c6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6 2023-04-06 05:56:54,711 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6, entries=150, sequenceid=298, filesize=25.4 K 2023-04-06 05:56:54,717 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/27d1234541724a249c673d1359754fcf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf 2023-04-06 05:56:54,717 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/cf556da6af564719a54ee12cbd4be3ac as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cf556da6af564719a54ee12cbd4be3ac 2023-04-06 05:56:54,721 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf, entries=150, sequenceid=298, filesize=11.9 K 2023-04-06 05:56:54,722 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/b92cbd2e601d46c18e011a458330196b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b 2023-04-06 05:56:54,722 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/A of fa1e1102963b7802ef3f913061227193 into cf556da6af564719a54ee12cbd4be3ac(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:54,722 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:54,722 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/A, priority=11, startTime=1680760613836; duration=0sec 2023-04-06 05:56:54,722 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:54,726 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b, entries=150, sequenceid=298, filesize=11.9 K 2023-04-06 05:56:54,726 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for fa1e1102963b7802ef3f913061227193 in 886ms, sequenceid=298, compaction requested=false 2023-04-06 05:56:54,727 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:54,727 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:54,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:54,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:54,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bdf71f0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:54,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:54,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:54,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:54,731 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:54,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:54,733 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:54,733 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:54,734 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:54,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:54,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:54,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:54,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,738 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:54,738 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:54,838 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:54,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:54,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:54,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:54,841 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:54,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:54,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:54,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:54,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:54,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:54,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:54,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:54,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:54,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:54,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:54,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:54,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:54,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:54,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:54,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:54,844 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:54,844 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:55,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:55,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760615130, max timestamp=1680760615140Num uniques 859; 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760615130, max timestamp=1680760615140Num uniques 859; 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760615130, max timestamp=1680760615140Num uniques 859; 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:55,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:56:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760675187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:55,263 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062f7471e9319a4710aae4eeb6dacdb506 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f7471e9319a4710aae4eeb6dacdb506 2023-04-06 05:56:55,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760675290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:55,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760675494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:55,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760675514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:55,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760675515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:55,671 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/70f214e7a8b74f028e87aeb63bba18f3 2023-04-06 05:56:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760675813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,081 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/831e96cb14a0494a8d173d7ea92a27ee 2023-04-06 05:56:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760676319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,490 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/06a9df5d31eb4e7a86c1bcc38c915af8 2023-04-06 05:56:56,494 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/70f214e7a8b74f028e87aeb63bba18f3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3 2023-04-06 05:56:56,497 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3, entries=150, sequenceid=312, filesize=25.4 K 2023-04-06 05:56:56,498 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/831e96cb14a0494a8d173d7ea92a27ee as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee 2023-04-06 05:56:56,501 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee, entries=150, sequenceid=312, filesize=11.9 K 2023-04-06 05:56:56,502 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/06a9df5d31eb4e7a86c1bcc38c915af8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8 2023-04-06 05:56:56,505 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8, entries=150, sequenceid=312, filesize=11.9 K 2023-04-06 05:56:56,506 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fa1e1102963b7802ef3f913061227193 in 1662ms, sequenceid=312, compaction requested=true 2023-04-06 05:56:56,506 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:56,506 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:56,507 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:56,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:56,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408b27d9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:56,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:56,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,510 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:56,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:56,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:56,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:56,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:56,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:56,513 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:56,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:56,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:56,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:56,513 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:56,514 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,514 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,514 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:56,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:56,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:56,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:56,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:56,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:56,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:56,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:56,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:56,520 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,520 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:56,520 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:56,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:56,620 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:56,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:56,623 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:56,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:56,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:56,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:56,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:56,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:56,625 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:56,625 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:56,625 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:56,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:56,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:56,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:56,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:56,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:56,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:56,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:56,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:56,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:56,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:56,626 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 05:56:56,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:56,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:56,627 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:56,627 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:56,627 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:56,627 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:56:57,040 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040621e98605280041b6b2b9707a066bf848 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040621e98605280041b6b2b9707a066bf848 2023-04-06 05:56:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:57,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:57,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:57,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760677366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,445 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/307cbcc84dd94c06840a62ac9a66c85e 2023-04-06 05:56:57,455 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/f6281e01730843fa838575e51172d5f3 2023-04-06 05:56:57,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:57,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760677468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59634 deadline: 1680760677523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,527 DEBUG [Thread-793] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:57,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760677535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,541 DEBUG [Thread-799] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:57,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760677672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,868 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/a64a0041c3d8462983beae98ebdabf12 2023-04-06 05:56:57,872 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/307cbcc84dd94c06840a62ac9a66c85e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e 2023-04-06 05:56:57,879 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e, entries=150, sequenceid=348, filesize=25.4 K 2023-04-06 05:56:57,880 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/f6281e01730843fa838575e51172d5f3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3 2023-04-06 05:56:57,885 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3, entries=150, sequenceid=348, filesize=11.9 K 2023-04-06 05:56:57,885 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/a64a0041c3d8462983beae98ebdabf12 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12 2023-04-06 05:56:57,891 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12, entries=150, sequenceid=348, filesize=11.9 K 2023-04-06 05:56:57,891 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for fa1e1102963b7802ef3f913061227193 in 1265ms, sequenceid=348, compaction requested=true 2023-04-06 05:56:57,891 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:57,891 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:57,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:57,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:57,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:57,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:57,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:57,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:57,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64911af[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:57,894 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:57,894 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:57,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:57,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:57,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:57,897 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:57,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:57,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:57,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:57,900 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:57,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:57,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:57,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:57,900 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:57,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:57,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:57,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:57,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:57,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:57,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:57,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:57,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:57,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:57,915 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:57,916 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:57,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:57,980 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:56:57,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:57,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:57,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:57,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:57,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:57,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:58,016 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,018 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,018 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,018 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,021 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,021 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,021 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,021 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,023 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4248fe2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,030 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,030 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,030 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,030 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,030 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:58,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760678034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,040 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,044 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,044 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,045 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760678135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,145 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,147 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,150 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,150 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,150 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,150 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302daa2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,153 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,156 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,156 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,157 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,157 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,157 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,157 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,160 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,160 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:58,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59702 deadline: 1680760678200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,204 DEBUG [Thread-791] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:56:58,260 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,263 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,265 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,265 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,266 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,266 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,266 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,266 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,268 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,268 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23791ebf[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,271 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,273 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,273 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,274 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,274 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,274 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,278 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,279 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,279 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:58,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760678340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,379 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,382 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,385 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,386 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,386 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,386 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,388 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@565b5ba1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,388 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,389 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,389 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,389 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,389 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,392 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,392 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,392 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,392 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,392 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,394 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040693ffb8105d2f45cf837635f93a2547a3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040693ffb8105d2f45cf837635f93a2547a3 2023-04-06 05:56:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,397 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,398 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,398 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,408 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/22eb9791b1534e1cbb1048a5d808cb99 2023-04-06 05:56:58,498 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,501 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,504 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,504 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,505 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,505 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,505 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,505 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,506 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,507 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582bf969[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,512 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,512 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,513 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,513 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,513 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,513 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,517 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,517 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,517 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,517 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,517 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,518 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,517 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,518 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,518 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,518 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,518 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,518 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,618 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,621 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,622 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,622 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,622 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,622 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,623 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,623 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,623 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,623 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,623 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,623 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,623 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,623 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,624 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,624 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,625 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4de9ef04[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,627 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,627 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,627 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,628 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,628 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,628 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,630 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,630 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,631 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,631 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:58,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,634 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,634 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:58,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760678645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,734 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,737 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,739 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,739 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:58,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:58,740 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:58,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@268e8ce0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:58,742 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:58,742 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,742 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:58,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:58,743 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:58,743 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,743 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:58,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,743 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,743 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,743 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:58,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:58,745 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:58,745 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:58,745 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:58,745 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:58,745 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:58,746 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,746 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:58,746 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:58,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:58,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:58,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:58,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:56:58,750 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:58,750 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:58,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/9c0ec6d3e8b84470a7e3a734f0c59c9b 2023-04-06 05:56:58,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/6ffb88e1dfb94540a816cf5f09c56a45 2023-04-06 05:56:58,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/22eb9791b1534e1cbb1048a5d808cb99 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99 2023-04-06 05:56:58,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99, entries=200, sequenceid=360, filesize=32.3 K 2023-04-06 05:56:58,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/9c0ec6d3e8b84470a7e3a734f0c59c9b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b 2023-04-06 05:56:58,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b, entries=150, sequenceid=360, filesize=11.9 K 2023-04-06 05:56:58,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/6ffb88e1dfb94540a816cf5f09c56a45 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45 2023-04-06 05:56:58,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45, entries=150, sequenceid=360, filesize=11.9 K 2023-04-06 05:56:58,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fa1e1102963b7802ef3f913061227193 in 869ms, sequenceid=360, compaction requested=true 2023-04-06 05:56:58,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:58,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:58,849 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:58,849 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:58,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:58,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:58,850 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:58,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:58,859 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:58,859 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/B is initiating minor compaction (all files) 2023-04-06 05:56:58,859 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/B in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,859 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6cbcb999735d4096ac2abb1189806290, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=60.0 K 2023-04-06 05:56:58,859 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:58,860 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/A is initiating minor compaction (all files) 2023-04-06 05:56:58,860 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/A in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,860 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cf556da6af564719a54ee12cbd4be3ac, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=134.5 K 2023-04-06 05:56:58,860 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6cbcb999735d4096ac2abb1189806290, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613306 2023-04-06 05:56:58,860 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 27d1234541724a249c673d1359754fcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680760613369 2023-04-06 05:56:58,861 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cf556da6af564719a54ee12cbd4be3ac, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613306 2023-04-06 05:56:58,862 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 831e96cb14a0494a8d173d7ea92a27ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680760613973 2023-04-06 05:56:58,862 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 8ba38c3cb7fd47619273f1f7e1aa77c6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680760613369 2023-04-06 05:56:58,862 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f6281e01730843fa838575e51172d5f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680760615154 2023-04-06 05:56:58,863 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 70f214e7a8b74f028e87aeb63bba18f3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680760613973 2023-04-06 05:56:58,863 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9c0ec6d3e8b84470a7e3a734f0c59c9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680760617359 2023-04-06 05:56:58,863 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 307cbcc84dd94c06840a62ac9a66c85e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680760615154 2023-04-06 05:56:58,864 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 22eb9791b1534e1cbb1048a5d808cb99, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680760617359 2023-04-06 05:56:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:58,866 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:58,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:58,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:58,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:58,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:58,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:58,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:58,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:58,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:58,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:58,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:58,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:58,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:58,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:58,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:58,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:58,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:58,875 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:56:58,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:58,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:58,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:58,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:58,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:58,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:58,888 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#B#compaction#157 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:58,896 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#A#compaction#156 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:59,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:56:59,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:56:59,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:59,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760679181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:59,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760679284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,314 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/0a7a843fec054619bec499aa07134a66 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a7a843fec054619bec499aa07134a66 2023-04-06 05:56:59,322 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/B of fa1e1102963b7802ef3f913061227193 into 0a7a843fec054619bec499aa07134a66(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:59,322 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:59,322 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/B, priority=11, startTime=1680760618849; duration=0sec 2023-04-06 05:56:59,323 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:56:59,323 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-06 05:56:59,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-06 05:56:59,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): fa1e1102963b7802ef3f913061227193/C is initiating minor compaction (all files) 2023-04-06 05:56:59,325 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fa1e1102963b7802ef3f913061227193/C in TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:59,325 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/e70c063d114e42af85305930beb84bb6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp, totalSize=60.0 K 2023-04-06 05:56:59,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e70c063d114e42af85305930beb84bb6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760613306 2023-04-06 05:56:59,326 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b92cbd2e601d46c18e011a458330196b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680760613369 2023-04-06 05:56:59,326 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 06a9df5d31eb4e7a86c1bcc38c915af8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680760613973 2023-04-06 05:56:59,326 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a64a0041c3d8462983beae98ebdabf12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680760615154 2023-04-06 05:56:59,327 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6ffb88e1dfb94540a816cf5f09c56a45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680760617359 2023-04-06 05:56:59,327 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065e7de30e013c4836aeb0134cb045114f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065e7de30e013c4836aeb0134cb045114f 2023-04-06 05:56:59,334 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=380, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/baa5eb19f0fa4f33b969a20f32e8a459 2023-04-06 05:56:59,339 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): fa1e1102963b7802ef3f913061227193#C#compaction#159 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:56:59,356 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d 2023-04-06 05:56:59,366 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/d5761b8d3f1a4598b1a5df364e675316 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/d5761b8d3f1a4598b1a5df364e675316 2023-04-06 05:56:59,373 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/C of fa1e1102963b7802ef3f913061227193 into d5761b8d3f1a4598b1a5df364e675316(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:56:59,373 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:59,373 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/C, priority=11, startTime=1680760618850; duration=0sec 2023-04-06 05:56:59,374 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:56:59,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:59,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760679486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,775 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/274f31d9b19d4a33a962d04e4ecfee38 2023-04-06 05:56:59,779 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/baa5eb19f0fa4f33b969a20f32e8a459 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa5eb19f0fa4f33b969a20f32e8a459 2023-04-06 05:56:59,783 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa5eb19f0fa4f33b969a20f32e8a459, entries=150, sequenceid=380, filesize=25.4 K 2023-04-06 05:56:59,784 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d 2023-04-06 05:56:59,788 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d, entries=150, sequenceid=380, filesize=11.9 K 2023-04-06 05:56:59,788 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/274f31d9b19d4a33a962d04e4ecfee38 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/274f31d9b19d4a33a962d04e4ecfee38 2023-04-06 05:56:59,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:56:59,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760679790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,795 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/274f31d9b19d4a33a962d04e4ecfee38, entries=150, sequenceid=380, filesize=11.9 K 2023-04-06 05:56:59,796 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fa1e1102963b7802ef3f913061227193 in 921ms, sequenceid=380, compaction requested=false 2023-04-06 05:56:59,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:56:59,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:59,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:56:59,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:56:59,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:56:59,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:56:59,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:59,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:56:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:59,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:56:59,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:56:59,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a844dc3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:56:59,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:56:59,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:56:59,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:56:59,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:56:59,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:56:59,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:59,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:56:59,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:56:59,814 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:56:59,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:56:59,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:56:59,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:56:59,814 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:56:59,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:56:59,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:56:59,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:56:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:56:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:56:59,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:56:59,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:56:59,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:56:59,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:56:59,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:56:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:56:59,819 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:56:59,819 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:56:59,919 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:56:59,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:56:59,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:56:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:56:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:56:59,922 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:56:59,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:59,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:56:59,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:56:59,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:59,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,923 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:56:59,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:56:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:56:59,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:56:59,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:56:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:59,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:56:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:56:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:56:59,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:56:59,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:56:59,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:56:59,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:56:59,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:56:59,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:56:59,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:56:59,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:56:59,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:56:59,925 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:56:59,925 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:56:59,936 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b47af99150364921ab40440928e40d22 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b47af99150364921ab40440928e40d22 2023-04-06 05:57:00,132 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/2a2652c240aa45f695c19ff39be59559 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/2a2652c240aa45f695c19ff39be59559 2023-04-06 05:57:00,147 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fa1e1102963b7802ef3f913061227193/A of fa1e1102963b7802ef3f913061227193 into 2a2652c240aa45f695c19ff39be59559(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 05:57:00,147 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:57:00,147 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., storeName=fa1e1102963b7802ef3f913061227193/A, priority=11, startTime=1680760618849; duration=1sec 2023-04-06 05:57:00,147 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:00,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:57:00,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:00,343 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=394, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8ee5b46ebd4f4b6ab93ac1de63c20985 2023-04-06 05:57:00,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760680364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:00,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760680467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:00,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760680670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:00,772 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/775df7c29ddf459785aa918ff138a500 2023-04-06 05:57:00,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760680977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,238 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/19b7bc43156d4436ae7d0cf3dcd41009 2023-04-06 05:57:01,253 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/8ee5b46ebd4f4b6ab93ac1de63c20985 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ee5b46ebd4f4b6ab93ac1de63c20985 2023-04-06 05:57:01,261 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ee5b46ebd4f4b6ab93ac1de63c20985, entries=150, sequenceid=394, filesize=25.4 K 2023-04-06 05:57:01,262 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/775df7c29ddf459785aa918ff138a500 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/775df7c29ddf459785aa918ff138a500 2023-04-06 05:57:01,271 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/775df7c29ddf459785aa918ff138a500, entries=150, sequenceid=394, filesize=11.9 K 2023-04-06 05:57:01,272 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/19b7bc43156d4436ae7d0cf3dcd41009 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/19b7bc43156d4436ae7d0cf3dcd41009 2023-04-06 05:57:01,281 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/19b7bc43156d4436ae7d0cf3dcd41009, entries=150, sequenceid=394, filesize=11.9 K 2023-04-06 05:57:01,283 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fa1e1102963b7802ef3f913061227193 in 1358ms, sequenceid=394, compaction requested=true 2023-04-06 05:57:01,283 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:57:01,283 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:01,283 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:01,283 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:01,283 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,284 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:01,284 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:01,285 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:01,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:01,285 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:01,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:01,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:01,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d63194b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:01,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:01,289 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,289 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:01,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:01,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:01,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:01,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:01,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:01,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:01,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:01,303 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:01,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:01,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:01,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:01,303 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:01,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:01,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:01,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:01,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:01,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:01,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:01,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,309 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:01,309 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:01,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:01,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:01,409 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:01,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:01,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:01,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:01,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:01,434 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:01,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:01,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:01,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:01,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:01,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:01,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:01,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:01,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:01,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:01,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:01,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:01,438 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:01,438 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:01,438 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:01,450 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:01,450 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:57:01,450 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:57:01,451 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:01,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:01,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:57:01,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760681539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59634 deadline: 1680760681540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:59684 deadline: 1680760681566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,570 DEBUG [Thread-799] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:57:01,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760681642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59634 deadline: 1680760681644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59634 deadline: 1680760681847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:01,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760681847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:01,881 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040694fb5cc4ac35473ab39f3385dbff7e1c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040694fb5cc4ac35473ab39f3385dbff7e1c 2023-04-06 05:57:02,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59634 deadline: 1680760682151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:02,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760682151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:02,297 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/6e9c0fa6da5a45efaf32b593374809c0 2023-04-06 05:57:02,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:02,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59664 deadline: 1680760682653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:59634 deadline: 1680760682657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:02,707 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/0a74386314d942a5aa3138d94081e9a2 2023-04-06 05:57:03,125 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/3d64a45f8861407f94b16b7bc8622d01 2023-04-06 05:57:03,129 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/6e9c0fa6da5a45efaf32b593374809c0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/6e9c0fa6da5a45efaf32b593374809c0 2023-04-06 05:57:03,132 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/6e9c0fa6da5a45efaf32b593374809c0, entries=150, sequenceid=415, filesize=25.4 K 2023-04-06 05:57:03,133 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/0a74386314d942a5aa3138d94081e9a2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a74386314d942a5aa3138d94081e9a2 2023-04-06 05:57:03,136 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a74386314d942a5aa3138d94081e9a2, entries=150, sequenceid=415, filesize=11.9 K 2023-04-06 05:57:03,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/3d64a45f8861407f94b16b7bc8622d01 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/3d64a45f8861407f94b16b7bc8622d01 2023-04-06 05:57:03,141 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/3d64a45f8861407f94b16b7bc8622d01, entries=150, sequenceid=415, filesize=11.9 K 2023-04-06 05:57:03,142 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fa1e1102963b7802ef3f913061227193 in 1692ms, sequenceid=415, compaction requested=true 2023-04-06 05:57:03,142 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:57:03,142 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:03,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:03,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:03,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:03,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:03,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:03,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff0422c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:03,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:03,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:03,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:03,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:03,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:03,146 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:03,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:03,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:03,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:03,148 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:03,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:03,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:03,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:03,148 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:03,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:03,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:03,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:03,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:03,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:03,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:03,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:03,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:03,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:03,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:03,152 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,152 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:03,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,252 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:03,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:03,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:03,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:03,255 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:03,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:03,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:03,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:03,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:03,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,256 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:03,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:03,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:03,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:03,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:03,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:03,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. started... 2023-04-06 05:57:03,258 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:57:03,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:03,471 DEBUG [Thread-804] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16505096 to 127.0.0.1:55165 2023-04-06 05:57:03,471 DEBUG [Thread-804] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,472 DEBUG [Thread-806] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03e1990e to 127.0.0.1:55165 2023-04-06 05:57:03,472 DEBUG [Thread-808] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d6367b2 to 127.0.0.1:55165 2023-04-06 05:57:03,472 DEBUG [Thread-806] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,472 DEBUG [Thread-808] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,473 DEBUG [Thread-802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7605c204 to 127.0.0.1:55165 2023-04-06 05:57:03,473 DEBUG [Thread-802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:03,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. as already flushing 2023-04-06 05:57:03,660 DEBUG [Thread-795] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61ad5d1c to 127.0.0.1:55165 2023-04-06 05:57:03,660 DEBUG [Thread-795] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,669 DEBUG [Thread-793] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x633c81ba to 127.0.0.1:55165 2023-04-06 05:57:03,669 DEBUG [Thread-793] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:03,674 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406cfa7bcbb289a4f81ba97fbc71db9a51c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cfa7bcbb289a4f81ba97fbc71db9a51c 2023-04-06 05:57:04,081 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/cbf062e5792847ccbe815907fc6cbef2 2023-04-06 05:57:04,501 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/774f46e83400415eb7c2e8abb1e56b17 2023-04-06 05:57:04,568 DEBUG [Thread-797] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43a14bcf to 127.0.0.1:55165 2023-04-06 05:57:04,568 DEBUG [Thread-797] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:04,918 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/9cbc46fe941548b89f837651cb5db0a8 2023-04-06 05:57:04,922 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/cbf062e5792847ccbe815907fc6cbef2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cbf062e5792847ccbe815907fc6cbef2 2023-04-06 05:57:04,927 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cbf062e5792847ccbe815907fc6cbef2, entries=150, sequenceid=427, filesize=25.4 K 2023-04-06 05:57:04,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/774f46e83400415eb7c2e8abb1e56b17 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/774f46e83400415eb7c2e8abb1e56b17 2023-04-06 05:57:04,936 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/774f46e83400415eb7c2e8abb1e56b17, entries=150, sequenceid=427, filesize=11.9 K 2023-04-06 05:57:04,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/9cbc46fe941548b89f837651cb5db0a8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/9cbc46fe941548b89f837651cb5db0a8 2023-04-06 05:57:04,944 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/9cbc46fe941548b89f837651cb5db0a8, entries=150, sequenceid=427, filesize=11.9 K 2023-04-06 05:57:04,945 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for fa1e1102963b7802ef3f913061227193 in 1687ms, sequenceid=427, compaction requested=true 2023-04-06 05:57:04,945 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:57:04,945 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:04,945 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:04,948 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:04,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:04,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:04,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dcb22be[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:04,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:04,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:04,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:04,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:04,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:04,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:04,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:04,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:04,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:04,959 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:04,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:04,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:04,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:04,959 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:04,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:04,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:04,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:04,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:04,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:04,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:04,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:04,983 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:04,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:04,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:04,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:04,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:04,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:04,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:04,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:04,984 DEBUG [Thread-801] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:04,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,984 DEBUG [Thread-801] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:04,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:04,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:04,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:04,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:05,084 DEBUG [Thread-801] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:05,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:08,227 DEBUG [Thread-791] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0df763e0 to 127.0.0.1:55165 2023-04-06 05:57:08,227 DEBUG [Thread-791] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:11,616 DEBUG [Thread-799] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6577ce5a to 127.0.0.1:55165 2023-04-06 05:57:11,617 DEBUG [Thread-799] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:11,617 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:57:11,617 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 56 2023-04-06 05:57:11,618 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-06 05:57:11,618 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 150 2023-04-06 05:57:11,618 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-06 05:57:11,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-06 05:57:11,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:57:11,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5162 2023-04-06 05:57:11,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5078 2023-04-06 05:57:11,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:57:11,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2152 2023-04-06 05:57:11,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 6452 rows 2023-04-06 05:57:11,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2164 2023-04-06 05:57:11,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 6491 rows 2023-04-06 05:57:11,621 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:57:11,622 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4894c837 to 127.0.0.1:55165 2023-04-06 05:57:11,622 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:11,624 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:57:11,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:57:11,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:11,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:11,628 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760631628"}]},"ts":"1680760631628"} 2023-04-06 05:57:11,629 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:57:11,641 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:57:11,642 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, UNASSIGN}] 2023-04-06 05:57:11,643 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, UNASSIGN 2023-04-06 05:57:11,644 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:11,644 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760631644"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760631644"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760631644"}]},"ts":"1680760631644"} 2023-04-06 05:57:11,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:11,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:11,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:11,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fa1e1102963b7802ef3f913061227193, disabling compactions & flushes 2023-04-06 05:57:11,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:11,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:11,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. after waiting 0 ms 2023-04-06 05:57:11,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:11,799 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing fa1e1102963b7802ef3f913061227193 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 05:57:11,799 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=A 2023-04-06 05:57:11,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:11,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=B 2023-04-06 05:57:11,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:11,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fa1e1102963b7802ef3f913061227193, store=C 2023-04-06 05:57:11,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:11,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:12,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040685056d911b19486d98fa61e81e2c406b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040685056d911b19486d98fa61e81e2c406b 2023-04-06 05:57:12,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=435, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/aca3795c9d3f4b6290b1824169e30e9a 2023-04-06 05:57:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:12,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/d6c8cc59df5d4f63b2b68f72491db7b1 2023-04-06 05:57:12,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/364f8377ef0d4f3a95f205abcc3c49a9 2023-04-06 05:57:12,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/A/aca3795c9d3f4b6290b1824169e30e9a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/aca3795c9d3f4b6290b1824169e30e9a 2023-04-06 05:57:12,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/aca3795c9d3f4b6290b1824169e30e9a, entries=150, sequenceid=435, filesize=25.4 K 2023-04-06 05:57:12,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/B/d6c8cc59df5d4f63b2b68f72491db7b1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d6c8cc59df5d4f63b2b68f72491db7b1 2023-04-06 05:57:12,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d6c8cc59df5d4f63b2b68f72491db7b1, entries=150, sequenceid=435, filesize=11.9 K 2023-04-06 05:57:12,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/.tmp/C/364f8377ef0d4f3a95f205abcc3c49a9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/364f8377ef0d4f3a95f205abcc3c49a9 2023-04-06 05:57:12,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/364f8377ef0d4f3a95f205abcc3c49a9, entries=150, sequenceid=435, filesize=11.9 K 2023-04-06 05:57:12,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for fa1e1102963b7802ef3f913061227193 in 882ms, sequenceid=435, compaction requested=true 2023-04-06 05:57:12,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/9c0963c5653041c5800638be28e8cff1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/d480fdc9acf645e6b6e37fc40eb30a0b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cf556da6af564719a54ee12cbd4be3ac, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99] to archive 2023-04-06 05:57:12,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:12,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/104c94854ad1484aa1132b81153dcac9 2023-04-06 05:57:12,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8d63eda6fb024c0884275b48b21b27c7 2023-04-06 05:57:12,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa1f302d66340d9873fec650c4217c3 2023-04-06 05:57:12,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/57176103b82e41ba9a4275ea87f3b7a4 2023-04-06 05:57:12,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/9c0963c5653041c5800638be28e8cff1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/9c0963c5653041c5800638be28e8cff1 2023-04-06 05:57:12,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/40ba052bb4984ad18c6df519a2db1a96 2023-04-06 05:57:12,710 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/684bb5a2d5d641709091a601f3ed59ef 2023-04-06 05:57:12,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/610dc541d6a84c6aa3a06ae1af82ba2d 2023-04-06 05:57:12,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/3b144295cb7a4c4aadc5216644598cdf 2023-04-06 05:57:12,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/d480fdc9acf645e6b6e37fc40eb30a0b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/d480fdc9acf645e6b6e37fc40eb30a0b 2023-04-06 05:57:12,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/f552d81fb50d4989a5ff2d81fba2a91a 2023-04-06 05:57:12,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/ed5b69e86c104a6e922945107a41aabf 2023-04-06 05:57:12,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/319fa6422dbe49f0a186d132508d163f 2023-04-06 05:57:12,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/deb912afaf1149d2bcea31e9e0ad227b 2023-04-06 05:57:12,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cf556da6af564719a54ee12cbd4be3ac to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cf556da6af564719a54ee12cbd4be3ac 2023-04-06 05:57:12,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ba38c3cb7fd47619273f1f7e1aa77c6 2023-04-06 05:57:12,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/70f214e7a8b74f028e87aeb63bba18f3 2023-04-06 05:57:12,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/307cbcc84dd94c06840a62ac9a66c85e 2023-04-06 05:57:12,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/22eb9791b1534e1cbb1048a5d808cb99 2023-04-06 05:57:12,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d206cff05ffe46ce8de87e2e661734b4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/7ffb816805d842f6948e374cacc8f00b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6cbcb999735d4096ac2abb1189806290, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b] to archive 2023-04-06 05:57:12,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:12,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/c453ccdde4a045c4b430a4a17440b369 2023-04-06 05:57:12,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:12,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/fe55e61316fe4af9bd055daf49f92922 2023-04-06 05:57:12,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/e752860b42694b539c5e5c211b797a47 2023-04-06 05:57:12,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d206cff05ffe46ce8de87e2e661734b4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d206cff05ffe46ce8de87e2e661734b4 2023-04-06 05:57:12,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/14e9c535f47c486a9c669df6d7078250 2023-04-06 05:57:12,738 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bbc925d548664f42b876700dff3091fc 2023-04-06 05:57:12,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5b738f66df2644a0810ee3d242bff237 2023-04-06 05:57:12,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6f28e5c047774ef0919c7603bb2c58cc 2023-04-06 05:57:12,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/7ffb816805d842f6948e374cacc8f00b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/7ffb816805d842f6948e374cacc8f00b 2023-04-06 05:57:12,741 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/bcc9d8ea2cc14f779495441de645f653 2023-04-06 05:57:12,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a599e584cab34f9c92777c5ae973a517 2023-04-06 05:57:12,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/911dd897f9874472b97a887f7d8dfe9b 2023-04-06 05:57:12,744 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/5f337ca647494f0195490c8c40f690d9 2023-04-06 05:57:12,744 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6cbcb999735d4096ac2abb1189806290 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/6cbcb999735d4096ac2abb1189806290 2023-04-06 05:57:12,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/a0828f1079e24b2f967ff8b467ee58b1 2023-04-06 05:57:12,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/27d1234541724a249c673d1359754fcf 2023-04-06 05:57:12,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/831e96cb14a0494a8d173d7ea92a27ee 2023-04-06 05:57:12,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/f6281e01730843fa838575e51172d5f3 2023-04-06 05:57:12,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/9c0ec6d3e8b84470a7e3a734f0c59c9b 2023-04-06 05:57:12,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/64a25352007140bea676645fc66c6273, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/4b40509f0ef6457489836d22160a982e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/e70c063d114e42af85305930beb84bb6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45] to archive 2023-04-06 05:57:12,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:12,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/03aec19faa074b13b0a7330e1aab6b61 2023-04-06 05:57:12,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/c3ea54bc32b54aa7a8693fc70a05ab3a 2023-04-06 05:57:12,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/ab43841f18ce49009e6b6f57c4940887 2023-04-06 05:57:12,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/64a25352007140bea676645fc66c6273 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/64a25352007140bea676645fc66c6273 2023-04-06 05:57:12,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/aa1a1f81e4124f9aacdec55a503e74d8 2023-04-06 05:57:12,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/2bde5518b99d41d3b5fb57aefba43f48 2023-04-06 05:57:12,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a763caaefec84e18bf91d8d09041c89f 2023-04-06 05:57:12,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/0a584eedba3d405e982eb04f96b0a866 2023-04-06 05:57:12,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/4b40509f0ef6457489836d22160a982e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/4b40509f0ef6457489836d22160a982e 2023-04-06 05:57:12,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/10c7e92cf95c47f0b321c255c21bb1bd 2023-04-06 05:57:12,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b6cd965409e14b6fbd17e376345e0cd8 2023-04-06 05:57:12,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/5d78ae6f393b4cd8b83b365fe48ef8cd 2023-04-06 05:57:12,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/04741216314f4f9380be55eacced84a9 2023-04-06 05:57:12,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/e70c063d114e42af85305930beb84bb6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/e70c063d114e42af85305930beb84bb6 2023-04-06 05:57:12,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6514349fb82443c19fd68c54978affe1 2023-04-06 05:57:12,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/b92cbd2e601d46c18e011a458330196b 2023-04-06 05:57:12,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/06a9df5d31eb4e7a86c1bcc38c915af8 2023-04-06 05:57:12,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/a64a0041c3d8462983beae98ebdabf12 2023-04-06 05:57:12,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/6ffb88e1dfb94540a816cf5f09c56a45 2023-04-06 05:57:12,783 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits/438.seqid, newMaxSeqId=438, maxSeqId=4 2023-04-06 05:57:12,785 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193. 2023-04-06 05:57:12,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fa1e1102963b7802ef3f913061227193: 2023-04-06 05:57:12,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:12,787 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=fa1e1102963b7802ef3f913061227193, regionState=CLOSED 2023-04-06 05:57:12,787 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760632787"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760632787"}]},"ts":"1680760632787"} 2023-04-06 05:57:12,789 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-06 05:57:12,789 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure fa1e1102963b7802ef3f913061227193, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.1430 sec 2023-04-06 05:57:12,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-06 05:57:12,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fa1e1102963b7802ef3f913061227193, UNASSIGN in 1.1470 sec 2023-04-06 05:57:12,791 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760632791"}]},"ts":"1680760632791"} 2023-04-06 05:57:12,792 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:57:12,793 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:57:12,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-06 05:57:13,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-06 05:57:13,734 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-06 05:57:13,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:57:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,737 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 05:57:13,738 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,742 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:13,744 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits] 2023-04-06 05:57:13,746 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/2a2652c240aa45f695c19ff39be59559 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/2a2652c240aa45f695c19ff39be59559 2023-04-06 05:57:13,748 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/6e9c0fa6da5a45efaf32b593374809c0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/6e9c0fa6da5a45efaf32b593374809c0 2023-04-06 05:57:13,749 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ee5b46ebd4f4b6ab93ac1de63c20985 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/8ee5b46ebd4f4b6ab93ac1de63c20985 2023-04-06 05:57:13,750 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/aca3795c9d3f4b6290b1824169e30e9a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/aca3795c9d3f4b6290b1824169e30e9a 2023-04-06 05:57:13,751 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa5eb19f0fa4f33b969a20f32e8a459 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/baa5eb19f0fa4f33b969a20f32e8a459 2023-04-06 05:57:13,752 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cbf062e5792847ccbe815907fc6cbef2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/A/cbf062e5792847ccbe815907fc6cbef2 2023-04-06 05:57:13,754 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a74386314d942a5aa3138d94081e9a2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a74386314d942a5aa3138d94081e9a2 2023-04-06 05:57:13,756 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a7a843fec054619bec499aa07134a66 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/0a7a843fec054619bec499aa07134a66 2023-04-06 05:57:13,757 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/774f46e83400415eb7c2e8abb1e56b17 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/774f46e83400415eb7c2e8abb1e56b17 2023-04-06 05:57:13,758 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/775df7c29ddf459785aa918ff138a500 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/775df7c29ddf459785aa918ff138a500 2023-04-06 05:57:13,759 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/b4e3b6f7ef374f13ac56d56d3d6cdf9d 2023-04-06 05:57:13,760 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d6c8cc59df5d4f63b2b68f72491db7b1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/B/d6c8cc59df5d4f63b2b68f72491db7b1 2023-04-06 05:57:13,762 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/19b7bc43156d4436ae7d0cf3dcd41009 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/19b7bc43156d4436ae7d0cf3dcd41009 2023-04-06 05:57:13,763 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/274f31d9b19d4a33a962d04e4ecfee38 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/274f31d9b19d4a33a962d04e4ecfee38 2023-04-06 05:57:13,764 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/364f8377ef0d4f3a95f205abcc3c49a9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/364f8377ef0d4f3a95f205abcc3c49a9 2023-04-06 05:57:13,765 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/3d64a45f8861407f94b16b7bc8622d01 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/3d64a45f8861407f94b16b7bc8622d01 2023-04-06 05:57:13,766 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/9cbc46fe941548b89f837651cb5db0a8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/9cbc46fe941548b89f837651cb5db0a8 2023-04-06 05:57:13,767 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/d5761b8d3f1a4598b1a5df364e675316 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/C/d5761b8d3f1a4598b1a5df364e675316 2023-04-06 05:57:13,768 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits/438.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193/recovered.edits/438.seqid 2023-04-06 05:57:13,770 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/fa1e1102963b7802ef3f913061227193 2023-04-06 05:57:13,770 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:57:13,770 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:57:13,771 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 05:57:13,774 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406083e282ad38e4b9bb9501229c56bb2a7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406083e282ad38e4b9bb9501229c56bb2a7 2023-04-06 05:57:13,775 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040613c569817a3c48a99cd5f890c45bae4a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040613c569817a3c48a99cd5f890c45bae4a 2023-04-06 05:57:13,775 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b3051180b4043b59594fae7d807439c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b3051180b4043b59594fae7d807439c 2023-04-06 05:57:13,776 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040621e98605280041b6b2b9707a066bf848 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040621e98605280041b6b2b9707a066bf848 2023-04-06 05:57:13,777 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062b2f68e4d7974ea2948f9593da20d288 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062b2f68e4d7974ea2948f9593da20d288 2023-04-06 05:57:13,778 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062dc42631c72a49908b1ad70495e32d1f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062dc42631c72a49908b1ad70495e32d1f 2023-04-06 05:57:13,779 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f24b87fe6964ab6ba7caf1c4a0928f6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f24b87fe6964ab6ba7caf1c4a0928f6 2023-04-06 05:57:13,780 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f7471e9319a4710aae4eeb6dacdb506 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062f7471e9319a4710aae4eeb6dacdb506 2023-04-06 05:57:13,781 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406369879dbc81c43adb4f7107d43f026c7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406369879dbc81c43adb4f7107d43f026c7 2023-04-06 05:57:13,783 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065e7de30e013c4836aeb0134cb045114f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065e7de30e013c4836aeb0134cb045114f 2023-04-06 05:57:13,784 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661ad3c243f4d449dad9843b868520df8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661ad3c243f4d449dad9843b868520df8 2023-04-06 05:57:13,785 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067c9828ed202244b3ba707bfcce50cff7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067c9828ed202244b3ba707bfcce50cff7 2023-04-06 05:57:13,787 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040685056d911b19486d98fa61e81e2c406b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040685056d911b19486d98fa61e81e2c406b 2023-04-06 05:57:13,788 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040693ffb8105d2f45cf837635f93a2547a3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040693ffb8105d2f45cf837635f93a2547a3 2023-04-06 05:57:13,789 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040694fb5cc4ac35473ab39f3385dbff7e1c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040694fb5cc4ac35473ab39f3385dbff7e1c 2023-04-06 05:57:13,790 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069f544f9fcb5e4d8bb941c864f06fddf4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069f544f9fcb5e4d8bb941c864f06fddf4 2023-04-06 05:57:13,792 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b0afb5e4518e47eaa1c4dd81960a7a7a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b0afb5e4518e47eaa1c4dd81960a7a7a 2023-04-06 05:57:13,793 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b47af99150364921ab40440928e40d22 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b47af99150364921ab40440928e40d22 2023-04-06 05:57:13,794 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cfa7bcbb289a4f81ba97fbc71db9a51c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406cfa7bcbb289a4f81ba97fbc71db9a51c 2023-04-06 05:57:13,796 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e60ba00138ab41009db82dc55cc232eb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e60ba00138ab41009db82dc55cc232eb 2023-04-06 05:57:13,797 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f704fe92393c42ddb7b49811f98e2cc5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f704fe92393c42ddb7b49811f98e2cc5 2023-04-06 05:57:13,798 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:57:13,800 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,803 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:57:13,805 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:57:13,807 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,807 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:57:13,807 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760633807"}]},"ts":"9223372036854775807"} 2023-04-06 05:57:13,810 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:57:13,810 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fa1e1102963b7802ef3f913061227193, NAME => 'TestAcidGuarantees,,1680760600301.fa1e1102963b7802ef3f913061227193.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:57:13,810 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:57:13,810 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760633810"}]},"ts":"9223372036854775807"} 2023-04-06 05:57:13,813 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:57:13,815 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,816 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 80 msec 2023-04-06 05:57:13,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-06 05:57:13,839 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-06 05:57:13,853 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=227 (was 236), OpenFileDescriptor=502 (was 556), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=272 (was 289), ProcessCount=169 (was 169), AvailableMemoryMB=3568 (was 3686) 2023-04-06 05:57:13,865 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=227, OpenFileDescriptor=502, MaxFileDescriptor=60000, SystemLoadAverage=272, ProcessCount=169, AvailableMemoryMB=3568 2023-04-06 05:57:13,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:57:13,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:57:13,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:13,870 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:57:13,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-06 05:57:13,871 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:57:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:13,872 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:13,872 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d empty. 2023-04-06 05:57:13,873 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:13,873 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:57:13,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:14,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:14,283 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:57:14,289 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 61b6df3b1818e224b3645def0a20728d, NAME => 'TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:57:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:14,696 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:57:14,697 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 61b6df3b1818e224b3645def0a20728d, disabling compactions & flushes 2023-04-06 05:57:14,697 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:14,697 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:14,697 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. after waiting 0 ms 2023-04-06 05:57:14,697 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:14,697 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:14,697 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:14,702 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:57:14,704 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760634704"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760634704"}]},"ts":"1680760634704"} 2023-04-06 05:57:14,706 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:57:14,707 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:57:14,707 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760634707"}]},"ts":"1680760634707"} 2023-04-06 05:57:14,708 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:57:14,711 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, ASSIGN}] 2023-04-06 05:57:14,713 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, ASSIGN 2023-04-06 05:57:14,714 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:57:14,866 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=61b6df3b1818e224b3645def0a20728d, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:14,867 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760634866"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760634866"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760634866"}]},"ts":"1680760634866"} 2023-04-06 05:57:14,869 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:14,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:15,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:15,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 61b6df3b1818e224b3645def0a20728d, NAME => 'TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:57:15,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:57:15,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,029 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,030 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A 2023-04-06 05:57:15,030 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A 2023-04-06 05:57:15,031 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:15,031 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61b6df3b1818e224b3645def0a20728d columnFamilyName A 2023-04-06 05:57:15,032 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(310): Store=61b6df3b1818e224b3645def0a20728d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:15,032 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,033 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B 2023-04-06 05:57:15,033 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B 2023-04-06 05:57:15,033 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:15,034 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61b6df3b1818e224b3645def0a20728d columnFamilyName B 2023-04-06 05:57:15,034 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(310): Store=61b6df3b1818e224b3645def0a20728d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:15,034 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,036 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C 2023-04-06 05:57:15,036 DEBUG [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C 2023-04-06 05:57:15,036 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:15,036 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61b6df3b1818e224b3645def0a20728d columnFamilyName C 2023-04-06 05:57:15,036 INFO [StoreOpener-61b6df3b1818e224b3645def0a20728d-1] regionserver.HStore(310): Store=61b6df3b1818e224b3645def0a20728d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:15,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:15,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:57:15,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:15,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:57:15,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 61b6df3b1818e224b3645def0a20728d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65059781, jitterRate=-0.030533716082572937}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:57:15,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:15,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., pid=30, masterSystemTime=1680760635022 2023-04-06 05:57:15,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:15,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:15,047 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=61b6df3b1818e224b3645def0a20728d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:15,047 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760635047"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760635047"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760635047"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760635047"}]},"ts":"1680760635047"} 2023-04-06 05:57:15,050 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-06 05:57:15,051 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 in 180 msec 2023-04-06 05:57:15,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-06 05:57:15,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, ASSIGN in 340 msec 2023-04-06 05:57:15,053 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:57:15,054 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760635053"}]},"ts":"1680760635053"} 2023-04-06 05:57:15,055 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:57:15,057 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:57:15,059 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1890 sec 2023-04-06 05:57:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-06 05:57:15,982 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-06 05:57:15,984 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7187fcf3 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:15,991 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d74611, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:15,993 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:15,995 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:15,996 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:57:15,999 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:57:16,001 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x712d78ca to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,031 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f837278, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,032 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x407b9cb0 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,037 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a8a575d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,038 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7f456fa2 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,136 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6347f2e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,137 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x62be7a0b to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,141 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36523181, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,141 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7abacd49 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,145 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b458e37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,145 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7bae1a64 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,149 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5d1710, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,149 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x5654f8ca to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,157 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d22a55e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,157 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x509adaa8 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,162 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2179ead9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,162 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7e5e7a93 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,169 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7897b06a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,170 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x3254ddef to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:16,173 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fd3a3d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:16,177 DEBUG [hconnection-0xd977221-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:16,177 DEBUG [hconnection-0x169ba5ca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,179 DEBUG [hconnection-0x48019643-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,181 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,182 DEBUG [hconnection-0x56ebcf8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,183 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,183 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:16,188 DEBUG [hconnection-0x21d70c88-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,188 DEBUG [hconnection-0x7cffd891-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,188 DEBUG [hconnection-0x7cff81e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,188 DEBUG [hconnection-0x389ca132-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,187 DEBUG [hconnection-0x347e43a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,190 DEBUG [hconnection-0x3c5ea08f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:16,190 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:16,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:16,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:16,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:16,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43858, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:16,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:16,193 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43874, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,195 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,195 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:16,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:16,196 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:16,197 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,199 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:16,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:16,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:16,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:16,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:16,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:16,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:16,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:16,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:16,216 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:16,216 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:16,216 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:16,216 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:16,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:16,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:16,218 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:16,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:16,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760696249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760696249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760696249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760696255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760696255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760696356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760696356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760696356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760696362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760696362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760696559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760696559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760696560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760696564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760696566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,704 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/348988be617c441097c5f1a6c05a345f 2023-04-06 05:57:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760696861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760696861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760696863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760696868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760696869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,115 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/82f65370af824c33a19c31db7c0938b7 2023-04-06 05:57:17,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:17,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760697364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:17,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760697366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:17,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760697367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760697370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760697371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,524 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/a84c69ba2d874c3f9c03ea8f136ff15e 2023-04-06 05:57:17,528 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/348988be617c441097c5f1a6c05a345f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f 2023-04-06 05:57:17,533 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f, entries=200, sequenceid=14, filesize=13.9 K 2023-04-06 05:57:17,533 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/82f65370af824c33a19c31db7c0938b7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7 2023-04-06 05:57:17,538 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7, entries=150, sequenceid=14, filesize=11.6 K 2023-04-06 05:57:17,539 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/a84c69ba2d874c3f9c03ea8f136ff15e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e 2023-04-06 05:57:17,544 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e, entries=150, sequenceid=14, filesize=11.6 K 2023-04-06 05:57:17,544 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 61b6df3b1818e224b3645def0a20728d in 1326ms, sequenceid=14, compaction requested=false 2023-04-06 05:57:17,544 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:17,544 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:17,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:17,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:17,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:17,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:17,545 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:17,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:17,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75298291[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:17,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:17,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:17,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:17,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:17,549 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,549 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:17,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:17,551 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:17,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:17,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:17,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:17,551 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:17,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:17,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:17,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:17,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:17,553 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:17,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:17,555 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:17,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:17,555 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:17,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,555 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:17,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:17,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:17,656 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:17,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:17,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:17,659 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:17,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:17,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:17,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:17,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:17,660 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,660 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:17,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:17,660 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:17,660 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:17,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:17,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:17,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:17,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:17,661 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:17,661 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:17,661 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:17,662 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:18,088 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/133b77d5d09e468784b54056ac2d473e 2023-04-06 05:57:18,316 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:57:18,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:18,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760698385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760698386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760698384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760698392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760698401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760698493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760698493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760698497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760698502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760698504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,540 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/3cc363061a664524b0410b9ce9fdacb9 2023-04-06 05:57:18,603 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/5852af8b4f744f968f5c8ec6d1ccf294 2023-04-06 05:57:18,607 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/133b77d5d09e468784b54056ac2d473e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e 2023-04-06 05:57:18,612 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 05:57:18,625 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/3cc363061a664524b0410b9ce9fdacb9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9 2023-04-06 05:57:18,640 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 05:57:18,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/5852af8b4f744f968f5c8ec6d1ccf294 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294 2023-04-06 05:57:18,650 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294, entries=150, sequenceid=34, filesize=11.6 K 2023-04-06 05:57:18,651 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 61b6df3b1818e224b3645def0a20728d in 990ms, sequenceid=34, compaction requested=false 2023-04-06 05:57:18,652 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:18,652 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:18,652 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:18,652 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:18,652 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,654 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:18,654 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:18,658 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:18,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b5743e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:18,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:18,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,668 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:18,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:18,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:18,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:18,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:18,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:18,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:18,674 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:18,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:18,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:18,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:18,674 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:18,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:18,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:18,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:18,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:18,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:18,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,688 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:18,689 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:18,689 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:18,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:57:18,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:18,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:18,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:18,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:18,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:18,771 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:18,789 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:18,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:18,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:18,793 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:18,793 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:18,793 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:18,793 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:18,794 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,794 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,794 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:18,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:18,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:18,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:18,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:18,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:18,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:18,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:18,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:18,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:18,815 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:18,815 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:18,815 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:18,815 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:18,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:18,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/19d942f384f94a80bbac365f82634bd0 2023-04-06 05:57:18,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:18,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df9af26[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:18,820 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:18,820 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:18,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:18,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:18,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:18,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:18,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:18,836 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:18,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:18,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:18,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:18,836 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:18,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:18,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:18,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:18,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:18,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:18,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760698838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760698838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,849 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:18,849 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:18,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:18,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:18,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:18,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:18,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:18,850 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:18,850 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:18,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:18,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/b5f89370cb3c4cf2aa42ce85d28bbcd6 2023-04-06 05:57:18,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:18,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760698843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,855 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:18,855 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760698851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760698851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,950 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:18,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:18,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760698949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760698955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760698958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760698961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760698967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:18,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:18,979 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:18,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:18,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:18,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:18,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:18,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:18,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:18,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:18,982 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,982 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:18,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:18,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:18,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:18,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:18,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:18,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:18,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:18,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:18,998 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:18,998 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:18,998 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:18,999 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:18,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:18,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:18,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:18,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:18,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,000 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4983aef5[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:19,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,006 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,006 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,008 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,008 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,008 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,009 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,013 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,014 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,014 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,114 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:19,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:19,152 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:19,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:19,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:19,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,153 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:19,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760699155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:19,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:19,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:19,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:19,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:19,167 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:19,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760699161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,167 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:19,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:19,168 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:19,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:19,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760699163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:19,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760699166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,169 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,169 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,169 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7007805e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,171 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,171 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,172 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,172 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,172 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,173 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,173 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760699171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:19,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,176 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,176 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,197 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,197 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:19,297 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:19,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:19,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:19,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:19,302 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:19,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:19,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:19,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:19,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:19,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:19,314 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:19,314 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,314 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:19,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:19,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:19,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:19,319 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:19,319 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/0775a5a68ef14c5296ad89e281949e8c 2023-04-06 05:57:19,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/19d942f384f94a80bbac365f82634bd0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0 2023-04-06 05:57:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b781ee4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 05:57:19,354 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/b5f89370cb3c4cf2aa42ce85d28bbcd6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6 2023-04-06 05:57:19,356 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,365 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,365 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 05:57:19,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/0775a5a68ef14c5296ad89e281949e8c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c 2023-04-06 05:57:19,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,378 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c, entries=150, sequenceid=50, filesize=11.6 K 2023-04-06 05:57:19,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 61b6df3b1818e224b3645def0a20728d in 615ms, sequenceid=50, compaction requested=true 2023-04-06 05:57:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:19,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,383 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:19,383 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:19,383 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:19,383 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:19,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,384 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:19,384 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:19,384 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:19,384 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:19,384 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,384 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,384 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=37.1 K 2023-04-06 05:57:19,384 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=34.8 K 2023-04-06 05:57:19,384 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 348988be617c441097c5f1a6c05a345f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760636206 2023-04-06 05:57:19,385 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 82f65370af824c33a19c31db7c0938b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760636206 2023-04-06 05:57:19,385 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 133b77d5d09e468784b54056ac2d473e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680760636247 2023-04-06 05:57:19,385 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3cc363061a664524b0410b9ce9fdacb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680760636247 2023-04-06 05:57:19,385 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 19d942f384f94a80bbac365f82634bd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:19,385 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b5f89370cb3c4cf2aa42ce85d28bbcd6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:19,400 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#183 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:19,403 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#184 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:19,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:19,466 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 05:57:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:19,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:19,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:19,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:19,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:19,483 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:19,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:19,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:19,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760699483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760699484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760699485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760699486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760699488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:19,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:19,495 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:19,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:19,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:19,498 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,498 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,500 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,500 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:19,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:19,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:19,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:19,501 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,501 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:19,501 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:19,501 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:19,501 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,504 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,505 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b580232[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,522 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,522 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:19,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,525 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,525 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,543 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,543 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,543 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,549 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,549 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760699588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760699590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760699590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760699592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760699593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,649 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:19,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:19,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:19,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:19,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:19,674 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:19,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:19,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:19,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:19,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:19,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:19,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:19,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:19,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:19,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:19,685 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:19,685 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:19,686 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,686 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:19,686 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:19,686 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:19,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f78e7aa[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,715 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:19,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,728 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,728 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,734 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,734 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760699792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760699792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760699793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760699796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:19,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760699796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,834 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:19,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:19,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:19,843 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:19,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:19,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:19,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:19,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:19,849 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/5753f9c3c6544c80bdf9279ec61e83b2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5753f9c3c6544c80bdf9279ec61e83b2 2023-04-06 05:57:19,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:19,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:19,849 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:19,849 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:19,849 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:19,849 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:19,849 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:19,849 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,850 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:19,850 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:19,850 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:19,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:19,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:19,850 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:19,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:19,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23319f6b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:19,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:19,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:19,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:19,871 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:19,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:19,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:19,875 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:19,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:19,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:19,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:19,876 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:19,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:19,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,878 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/7ae63179354944369cef4d39f7f27598 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/7ae63179354944369cef4d39f7f27598 2023-04-06 05:57:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:19,903 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into 5753f9c3c6544c80bdf9279ec61e83b2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:19,903 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:19,903 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=13, startTime=1680760639383; duration=0sec 2023-04-06 05:57:19,903 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:19,903 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:19,904 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into 7ae63179354944369cef4d39f7f27598(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:19,904 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:19,904 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=13, startTime=1680760639383; duration=0sec 2023-04-06 05:57:19,905 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:19,908 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:19,908 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,909 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:19,909 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=34.8 K 2023-04-06 05:57:19,909 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a84c69ba2d874c3f9c03ea8f136ff15e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760636206 2023-04-06 05:57:19,909 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5852af8b4f744f968f5c8ec6d1ccf294, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680760636247 2023-04-06 05:57:19,910 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 0775a5a68ef14c5296ad89e281949e8c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:19,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:19,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:19,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:19,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:19,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:19,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:19,915 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:19,915 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:19,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:19,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:19,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:19,918 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:19,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:19,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3cb8b281a66b4a86aab92611ceb8df48 2023-04-06 05:57:19,929 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:19,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/b7e7144abc5141b68b2c5005fa4fda61 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b7e7144abc5141b68b2c5005fa4fda61 2023-04-06 05:57:19,973 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into b7e7144abc5141b68b2c5005fa4fda61(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:19,973 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:19,973 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=13, startTime=1680760639383; duration=0sec 2023-04-06 05:57:19,973 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:20,017 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,020 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,024 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,024 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,024 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,025 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,025 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,036 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,036 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78652ad7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,040 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,041 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,041 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,041 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,041 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,042 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,042 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,044 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,044 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,046 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,046 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,046 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,063 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,063 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,063 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,064 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,064 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,064 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,064 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760700095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760700097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760700097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760700099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760700102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,163 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,166 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,171 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,171 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,171 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,171 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,172 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357ee7b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,195 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,199 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,199 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,201 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,201 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,201 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,203 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,207 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,207 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:20,314 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,325 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,340 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bba0c0c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,349 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,349 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,349 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,350 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,353 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,353 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,353 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,353 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,353 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/514e59e07d234664ad97cb1367a57890 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,373 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,373 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:20,474 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,485 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,508 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,508 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,508 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,508 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df04d7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,512 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,512 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,512 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,513 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,513 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,521 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,521 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:20,533 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,533 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:20,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760700598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760700600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760700601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760700610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:20,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760700611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,633 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,637 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,638 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,640 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,640 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,640 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,640 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@553db509[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,657 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,661 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,671 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,671 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,672 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,674 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,685 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:20,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,685 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,686 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,686 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,786 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,789 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,794 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,806 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,806 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,806 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:20,806 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:20,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/691c2905c39c40dc8f3578720f164f96 2023-04-06 05:57:20,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,810 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3cb8b281a66b4a86aab92611ceb8df48 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48 2023-04-06 05:57:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:20,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6925ade3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:20,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:20,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:57:20,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,826 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:20,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:20,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/514e59e07d234664ad97cb1367a57890 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890 2023-04-06 05:57:20,826 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:20,826 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:20,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:20,827 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:20,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:20,830 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:20,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:20,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:20,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:20,830 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:20,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:57:20,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:20,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/691c2905c39c40dc8f3578720f164f96 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96, entries=150, sequenceid=68, filesize=11.6 K 2023-04-06 05:57:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 61b6df3b1818e224b3645def0a20728d in 1384ms, sequenceid=68, compaction requested=false 2023-04-06 05:57:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:20,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,851 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:20,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:20,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:20,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:20,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,852 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:20,853 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:20,853 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:20,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:20,953 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:20,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:20,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:20,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:20,961 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:20,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:20,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:20,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:20,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:20,966 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:20,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:20,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:20,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:20,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:20,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:20,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:20,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:20,974 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:20,974 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:20,974 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:20,974 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:20,995 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/07eb3436b2d343d9814865d1192c32b1 2023-04-06 05:57:21,432 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/372a0e38e85848c6bcd72648e8263eb8 2023-04-06 05:57:21,460 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4e6ee0880e1f4ecea130824df24e6646 2023-04-06 05:57:21,464 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/07eb3436b2d343d9814865d1192c32b1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1 2023-04-06 05:57:21,467 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1, entries=150, sequenceid=85, filesize=11.6 K 2023-04-06 05:57:21,468 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/372a0e38e85848c6bcd72648e8263eb8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8 2023-04-06 05:57:21,472 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8, entries=150, sequenceid=85, filesize=11.6 K 2023-04-06 05:57:21,475 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4e6ee0880e1f4ecea130824df24e6646 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646 2023-04-06 05:57:21,484 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646, entries=150, sequenceid=85, filesize=11.6 K 2023-04-06 05:57:21,485 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 61b6df3b1818e224b3645def0a20728d in 511ms, sequenceid=85, compaction requested=true 2023-04-06 05:57:21,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:21,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:21,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120416a7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:21,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:21,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:21,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:21,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:21,489 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:21,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:21,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:21,501 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:21,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:21,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:21,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:21,501 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:21,507 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:21,511 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:21,511 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:21,507 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:21,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,612 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:21,612 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:21,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:21,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:21,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:21,627 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:21,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:21,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:21,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:21,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760641624, max timestamp=1680760641642Num uniques 859; 2023-04-06 05:57:21,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:21,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760641624, max timestamp=1680760641642Num uniques 859; 2023-04-06 05:57:21,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:57:21,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680760641624, max timestamp=1680760641642Num uniques 859; 2023-04-06 05:57:21,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-06 05:57:21,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:21,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/8d4ca8919828414a932a4e93c772ee8f 2023-04-06 05:57:21,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:21,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:21,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:21,655 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:21,655 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:21,655 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:21,655 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:21,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:21,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,660 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:21,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e81418c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:21,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:21,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:21,675 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:21,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,676 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:21,678 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760701677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760701676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760701677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760701679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760701680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:21,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:21,694 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:21,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:21,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:21,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:21,694 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:21,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,696 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:21,696 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,696 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:21,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,700 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:21,700 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:21,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/093a9303526b40a89e450b55802295d5 2023-04-06 05:57:21,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760701781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760701781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760701784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760701786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760701787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,800 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:21,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:21,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:21,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:21,806 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:21,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:21,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:21,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:21,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,810 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:21,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:21,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:21,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:21,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:21,815 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:21,822 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:21,822 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:21,822 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:21,822 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:21,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:21,830 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:21,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ac34a8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:21,834 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:21,834 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,841 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:21,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:21,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:21,842 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:21,842 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,842 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:21,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:21,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:21,847 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:21,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:21,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:21,848 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:21,848 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:21,849 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,849 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:21,849 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:21,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,851 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:21,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:21,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:21,874 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:21,875 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:21,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:21,975 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:21,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:21,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760701989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760701990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760701990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760701990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:21,993 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:21,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:21,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760701992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:21,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:21,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:21,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:21,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,010 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a2c12b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,018 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,018 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,029 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,029 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,038 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,049 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,049 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,050 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/b73b2c43b3b74787afe9195af29ad8af 2023-04-06 05:57:22,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/8d4ca8919828414a932a4e93c772ee8f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f 2023-04-06 05:57:22,149 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f, entries=150, sequenceid=95, filesize=11.6 K 2023-04-06 05:57:22,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/093a9303526b40a89e450b55802295d5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5 2023-04-06 05:57:22,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,160 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5, entries=150, sequenceid=95, filesize=11.6 K 2023-04-06 05:57:22,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,169 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,170 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,170 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,170 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,170 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,170 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/b73b2c43b3b74787afe9195af29ad8af as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af 2023-04-06 05:57:22,176 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,177 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,177 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a15cf6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af, entries=150, sequenceid=95, filesize=11.6 K 2023-04-06 05:57:22,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.39 KB/136590 for 61b6df3b1818e224b3645def0a20728d in 569ms, sequenceid=95, compaction requested=true 2023-04-06 05:57:22,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:22,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:22,182 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:22,182 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:22,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:22,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:22,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,183 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:22,183 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:22,183 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,183 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,183 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:22,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,183 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:22,183 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,183 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5753f9c3c6544c80bdf9279ec61e83b2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=46.5 K 2023-04-06 05:57:22,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,183 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/7ae63179354944369cef4d39f7f27598, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=46.5 K 2023-04-06 05:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,184 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5753f9c3c6544c80bdf9279ec61e83b2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:22,184 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 7ae63179354944369cef4d39f7f27598, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:22,184 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 514e59e07d234664ad97cb1367a57890, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760638833 2023-04-06 05:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3cb8b281a66b4a86aab92611ceb8df48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760638833 2023-04-06 05:57:22,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 372a0e38e85848c6bcd72648e8263eb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680760639477 2023-04-06 05:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 093a9303526b40a89e450b55802295d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:22,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 07eb3436b2d343d9814865d1192c32b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680760639477 2023-04-06 05:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 8d4ca8919828414a932a4e93c772ee8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,187 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,187 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,190 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,191 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,191 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,192 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,192 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,223 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#195 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:22,227 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#196 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:22,291 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=140.10 KB heapSize=403.78 KB 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:22,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:57:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:22,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,300 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/ea770357afe942f0af7ebf99612b2d03 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/ea770357afe942f0af7ebf99612b2d03 2023-04-06 05:57:22,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760702303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760702307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,318 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into ea770357afe942f0af7ebf99612b2d03(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:22,318 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:22,318 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=12, startTime=1680760642182; duration=0sec 2023-04-06 05:57:22,318 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:22,318 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:22,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,321 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,322 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:22,322 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:22,322 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,322 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b7e7144abc5141b68b2c5005fa4fda61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=46.5 K 2023-04-06 05:57:22,322 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting b7e7144abc5141b68b2c5005fa4fda61, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680760638702 2023-04-06 05:57:22,323 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 691c2905c39c40dc8f3578720f164f96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1680760638833 2023-04-06 05:57:22,323 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,323 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4e6ee0880e1f4ecea130824df24e6646, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680760639477 2023-04-06 05:57:22,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,323 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting b73b2c43b3b74787afe9195af29ad8af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:22,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760702314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760702314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760702315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,340 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#198 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,342 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,344 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,344 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,344 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,344 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,344 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,359 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,359 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,360 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c8e44f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,363 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,364 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,365 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,365 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,365 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,366 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/232bc8f1d0e840d898adfac869e5500d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/232bc8f1d0e840d898adfac869e5500d 2023-04-06 05:57:22,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,375 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,375 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,380 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,380 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,380 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,380 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,381 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,381 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,387 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into 232bc8f1d0e840d898adfac869e5500d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:22,387 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:22,387 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=12, startTime=1680760642182; duration=0sec 2023-04-06 05:57:22,387 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:22,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760702409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760702425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,481 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,485 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,500 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,500 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,500 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,500 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,504 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,504 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73f6910a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,505 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,518 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,518 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,518 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,523 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,525 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,526 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760702612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,626 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760702629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,631 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,632 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,636 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7759adc6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,638 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,639 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,641 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,641 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,645 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,645 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,679 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/41eec27da08b45e4a01db217a771998f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/41eec27da08b45e4a01db217a771998f 2023-04-06 05:57:22,684 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into 41eec27da08b45e4a01db217a771998f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:22,684 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:22,684 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=12, startTime=1680760642182; duration=0sec 2023-04-06 05:57:22,684 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:22,745 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,749 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3e89981ff0f3463abceb03fcebba63c8 2023-04-06 05:57:22,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,764 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,770 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,770 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,770 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,770 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,771 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db50937[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,773 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,773 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,778 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,787 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,787 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,787 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,787 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,787 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,793 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,793 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,793 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,801 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:22,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,809 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,809 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760702810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760702828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760702828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,909 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:22,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:22,913 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:22,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:22,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:22,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,915 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:22,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:22,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:22,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:22,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:22,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:22,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:22,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:22,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:22,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:22,920 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:22,920 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:22,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760702922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,926 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,926 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:22,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e5c6452[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:22,928 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:22,928 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:22,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:22,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:22,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:22,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760702932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:22,938 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:22,938 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:22,938 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:22,938 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:22,938 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:22,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:22,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:22,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:22,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:22,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:22,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:22,960 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:22,960 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:22,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:22,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:22,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,061 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:23,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:23,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:23,064 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:23,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:23,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:23,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:23,066 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:23,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:23,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:23,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:23,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:23,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:23,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:23,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,104 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:23,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e5c052[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:23,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:23,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,110 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:23,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:23,111 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:23,111 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,111 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:23,117 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:23,119 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:23,119 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:23,119 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:23,119 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:23,120 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:23,130 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,130 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,131 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:23,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:23,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:23,146 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:23,146 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:23,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/c3feda578a444c13bfcbf07dfffd49b9 2023-04-06 05:57:23,246 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:23,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:23,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:23,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:23,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:23,249 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:23,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:23,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:23,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,254 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,254 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:23,251 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:23,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:23,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:23,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:23,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:23,270 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,270 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:23,270 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:23,270 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:23,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:23,271 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23686935[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:23,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:23,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:23,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:23,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:23,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:23,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:23,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:23,279 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:23,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:23,284 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:23,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:23,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:23,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:23,284 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:23,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:23,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:23,304 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:23,304 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:23,404 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:23,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:23,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:23,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:23,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:23,409 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:23,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:23,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:23,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,410 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:23,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:23,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:23,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:23,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:23,418 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:23,419 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,419 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:23,419 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:23,419 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:23,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:23,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760703424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:23,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56e130de[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:23,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:23,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,435 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:23,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:23,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:23,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,437 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:23,439 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:23,439 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:23,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760703437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:23,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:23,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:23,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,457 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,457 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:23,557 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:23,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:23,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:23,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:23,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:23,560 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:23,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:23,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:23,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:23,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:23,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:23,582 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:23,582 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:23,582 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:23,582 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:23,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:23,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:23,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba2a0a4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:23,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:23,587 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,587 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,587 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:23,587 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:23,587 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:23,587 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,588 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:23,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,588 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:23,588 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,588 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:23,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:23,597 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:23,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:23,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:23,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:23,597 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:23,603 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,603 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:23,603 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/ba0bf5463fa54fc4ac1815ead016a344 2023-04-06 05:57:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,616 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3e89981ff0f3463abceb03fcebba63c8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8 2023-04-06 05:57:23,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,618 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:23,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:23,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:23,618 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:23,618 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:23,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8, entries=150, sequenceid=133, filesize=11.7 K 2023-04-06 05:57:23,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/c3feda578a444c13bfcbf07dfffd49b9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9 2023-04-06 05:57:23,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9, entries=150, sequenceid=133, filesize=11.7 K 2023-04-06 05:57:23,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/ba0bf5463fa54fc4ac1815ead016a344 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344 2023-04-06 05:57:23,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344, entries=150, sequenceid=133, filesize=11.7 K 2023-04-06 05:57:23,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.10 KB/143460, heapSize ~403.73 KB/413424, currentSize=46.96 KB/48090 for 61b6df3b1818e224b3645def0a20728d in 1335ms, sequenceid=133, compaction requested=false 2023-04-06 05:57:23,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:23,718 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:23,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:23,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:23,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:23,721 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:23,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:23,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:23,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:23,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:23,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:23,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:23,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:23,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:23,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:23,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:23,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:23,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:23,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:23,733 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:23,733 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:23,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:23,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760643837, max timestamp=1680760643855Num uniques 859; 2023-04-06 05:57:23,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:23,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760643837, max timestamp=1680760643855Num uniques 859; 2023-04-06 05:57:23,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760643837, max timestamp=1680760643855Num uniques 859; 2023-04-06 05:57:23,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:23,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760703876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760703876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760703880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760703981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760703982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:23,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760703982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,156 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cc9fdfed40344f2e817152ea0cba8f37 2023-04-06 05:57:24,175 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/5d5f7d4c455241628ef264045d058d58 2023-04-06 05:57:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760704184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760704185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760704185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,212 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/d20db5f7e8fb469bb0e4aa75eeb8d02e 2023-04-06 05:57:24,216 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cc9fdfed40344f2e817152ea0cba8f37 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37 2023-04-06 05:57:24,219 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 05:57:24,220 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/5d5f7d4c455241628ef264045d058d58 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58 2023-04-06 05:57:24,239 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 05:57:24,245 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/d20db5f7e8fb469bb0e4aa75eeb8d02e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e 2023-04-06 05:57:24,255 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 05:57:24,257 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 61b6df3b1818e224b3645def0a20728d in 524ms, sequenceid=146, compaction requested=true 2023-04-06 05:57:24,257 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:24,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:24,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:24,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:24,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:24,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c720ad2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:24,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:24,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:24,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:24,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:24,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:24,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:24,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:24,278 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:24,278 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:24,278 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:24,278 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:24,278 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:24,279 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,279 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:24,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,279 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:24,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:24,280 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:24,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:24,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:24,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:24,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:24,294 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:24,294 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:24,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,295 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:24,295 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,394 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:24,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:24,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:24,402 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:24,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:24,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:24,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:24,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:24,403 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:24,403 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:24,404 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,404 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:24,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:24,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:24,404 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:24,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:24,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:24,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:24,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:24,405 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:57:24,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:24,406 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:24,406 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:24,406 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:24,406 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:24,406 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:24,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:24,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:24,450 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/a493720c76fa4786b0b78b4674e194a7 2023-04-06 05:57:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760704468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760704468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760704488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760704495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760704495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760704569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760704569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760704776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760704777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:24,870 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/acac03d75fac4a529cfc09bf89b6ca54 2023-04-06 05:57:24,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:24,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760704994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760705005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760705013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760705083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760705083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/957fe62627534dc09c6f26eec842765b 2023-04-06 05:57:25,303 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/a493720c76fa4786b0b78b4674e194a7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7 2023-04-06 05:57:25,307 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7, entries=150, sequenceid=182, filesize=11.8 K 2023-04-06 05:57:25,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/acac03d75fac4a529cfc09bf89b6ca54 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54 2023-04-06 05:57:25,312 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54, entries=150, sequenceid=182, filesize=11.8 K 2023-04-06 05:57:25,313 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/957fe62627534dc09c6f26eec842765b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b 2023-04-06 05:57:25,319 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b, entries=150, sequenceid=182, filesize=11.8 K 2023-04-06 05:57:25,319 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 61b6df3b1818e224b3645def0a20728d in 914ms, sequenceid=182, compaction requested=true 2023-04-06 05:57:25,320 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:25,320 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:25,320 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:25,320 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:25,320 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:25,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:25,324 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:25,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb3f1a8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:25,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:25,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:25,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:25,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:25,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:25,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:25,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:25,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:25,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:25,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:25,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:25,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:25,330 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:25,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:25,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:25,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:25,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:25,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:25,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:25,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:25,335 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:25,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:25,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,335 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:25,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:25,436 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:25,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:25,441 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:25,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:25,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:25,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:25,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:25,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:25,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:25,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:25,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:25,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:25,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:25,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:25,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:25,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:25,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:25,452 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:25,452 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:25,453 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:25,453 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:25,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760645614, max timestamp=1680760645632Num uniques 859; 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760645614, max timestamp=1680760645632Num uniques 859; 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760645614, max timestamp=1680760645632Num uniques 859; 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:25,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:25,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760705672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760705676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760705777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760705791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,888 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cfa79df4b00f464a9fccf6f0fbab5006 2023-04-06 05:57:25,937 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2ff6b9f8843d4220a716a2cc9c106039 2023-04-06 05:57:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760705983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:25,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760705995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760706002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760706018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760706021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760706288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760706302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,368 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/13ffd61d76be4c039d998b93d4404d9d 2023-04-06 05:57:26,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cfa79df4b00f464a9fccf6f0fbab5006 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006 2023-04-06 05:57:26,408 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006, entries=150, sequenceid=192, filesize=11.8 K 2023-04-06 05:57:26,409 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2ff6b9f8843d4220a716a2cc9c106039 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039 2023-04-06 05:57:26,419 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039, entries=150, sequenceid=192, filesize=11.8 K 2023-04-06 05:57:26,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/13ffd61d76be4c039d998b93d4404d9d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d 2023-04-06 05:57:26,455 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d, entries=150, sequenceid=192, filesize=11.8 K 2023-04-06 05:57:26,456 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 61b6df3b1818e224b3645def0a20728d in 1004ms, sequenceid=192, compaction requested=true 2023-04-06 05:57:26,456 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:26,456 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:26,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:26,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:26,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:26,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:26,462 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,462 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:26,463 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:26,463 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:26,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630b5120[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:26,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:26,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:26,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:26,495 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:26,495 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:26,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,495 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:26,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:26,497 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:26,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:26,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:26,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:26,497 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:26,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:26,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:26,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,500 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,505 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:26,505 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:26,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:26,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,506 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:26,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:26,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:26,507 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:26,507 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:26,607 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:26,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:26,626 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:26,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:26,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:26,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:26,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:26,629 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:26,629 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:26,629 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,629 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:26,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:26,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:26,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:26,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:26,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:26,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:26,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:26,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:26,642 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:57:26,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:26,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:26,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:26,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:26,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:26,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:26,679 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/4d802145d40b459797f6b2f6f1c50410 2023-04-06 05:57:26,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:26,797 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:26,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760706829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760706834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760706936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:26,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760706936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,141 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/9755ff7673bb42e4b9a1a0f2dbaf2992 2023-04-06 05:57:27,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760707140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:27,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760707144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:27,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43906 deadline: 1680760707450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43874 deadline: 1680760707453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,572 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/a1bbed09903945d7947fafe35e86fb88 2023-04-06 05:57:27,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/4d802145d40b459797f6b2f6f1c50410 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410 2023-04-06 05:57:27,583 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410, entries=150, sequenceid=228, filesize=11.8 K 2023-04-06 05:57:27,584 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/9755ff7673bb42e4b9a1a0f2dbaf2992 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992 2023-04-06 05:57:27,594 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992, entries=150, sequenceid=228, filesize=11.8 K 2023-04-06 05:57:27,597 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/a1bbed09903945d7947fafe35e86fb88 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88 2023-04-06 05:57:27,604 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88, entries=150, sequenceid=228, filesize=11.8 K 2023-04-06 05:57:27,605 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 61b6df3b1818e224b3645def0a20728d in 963ms, sequenceid=228, compaction requested=true 2023-04-06 05:57:27,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:27,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:27,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:27,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:27,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:27,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:27,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:27,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:27,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:27,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:27,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d526a65[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:27,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:27,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:27,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:27,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:27,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:27,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:27,611 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:27,611 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:27,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:27,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:27,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:27,614 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:27,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:27,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:27,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:27,614 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:27,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:27,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:27,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,618 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:27,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:27,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:27,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:27,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:27,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:27,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,635 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:27,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:27,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:27,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:27,635 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:27,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:27,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:27,635 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:27,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:27,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:27,735 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:27,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:27,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:27,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:27,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:27,744 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:27,745 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:27,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:27,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:27,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:27,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:27,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:27,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:27,747 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:27,747 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:27,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:27,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:27,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:27,775 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/e06270c1541048df86c2f37ac6a8d5da 2023-04-06 05:57:27,793 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/1ae9f105826e4981ad736d29784e1f0a 2023-04-06 05:57:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:27,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:28,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760648009, max timestamp=1680760648042Num uniques 859; 2023-04-06 05:57:28,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:28,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760648009, max timestamp=1680760648042Num uniques 859; 2023-04-06 05:57:28,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:28,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760648009, max timestamp=1680760648042Num uniques 859; 2023-04-06 05:57:28,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760708068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760708066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760708072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760708073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760708073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760708174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760708174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760708182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760708185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760708187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,211 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/91eb9bb125c74789b1f3848f320b7521 2023-04-06 05:57:28,229 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/e06270c1541048df86c2f37ac6a8d5da as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da 2023-04-06 05:57:28,239 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da, entries=100, sequenceid=238, filesize=9.4 K 2023-04-06 05:57:28,240 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/1ae9f105826e4981ad736d29784e1f0a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a 2023-04-06 05:57:28,248 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a, entries=100, sequenceid=238, filesize=9.4 K 2023-04-06 05:57:28,250 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/91eb9bb125c74789b1f3848f320b7521 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521 2023-04-06 05:57:28,262 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521, entries=100, sequenceid=238, filesize=9.4 K 2023-04-06 05:57:28,263 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 61b6df3b1818e224b3645def0a20728d in 516ms, sequenceid=238, compaction requested=true 2023-04-06 05:57:28,263 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:28,263 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:28,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:28,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:28,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:28,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:28,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513ad787[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:28,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:28,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,286 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,286 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,286 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:28,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:28,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:28,286 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:28,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:28,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:28,292 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:28,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:28,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:28,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:28,292 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:28,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,298 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:28,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:28,298 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:28,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:28,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,308 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:28,308 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:28,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:28,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-06 05:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:28,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:28,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:28,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:28,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:28,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:28,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760708395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760708396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760708400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,408 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:28,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760708406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:28,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760708409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:28,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:28,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:28,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:28,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,424 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:28,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:28,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:28,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:28,432 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:28,432 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,432 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:28,432 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:28,432 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:28,433 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:28,435 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:28,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a752b9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:28,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:28,453 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,454 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:28,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:28,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:28,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:28,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:28,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:28,460 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:28,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:28,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:28,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:28,460 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:28,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,470 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,483 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,483 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:28,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,484 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:28,484 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:28,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760708507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760708507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760708511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760708515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760708516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,584 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:28,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:28,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:28,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:28,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:28,592 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:28,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:28,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:28,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:28,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:28,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:28,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:28,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:28,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:28,599 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,599 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:28,599 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:28,599 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:28,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:28,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,607 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:28,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9c9f9c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:28,619 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:28,619 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,620 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:28,621 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:28,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:28,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:28,627 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:28,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:28,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:28,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:28,627 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:28,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,629 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:28,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:28,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,633 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:28,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,634 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,633 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:28,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760708717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760708717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760708720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760708726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760708727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,734 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:28,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:28,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:28,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:28,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:28,743 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:28,743 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,743 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:28,743 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:28,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,748 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:28,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:28,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:28,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:28,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:28,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:28,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:28,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,764 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,764 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:28,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4513d5c8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:28,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:28,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:28,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:28,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:28,770 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:28,771 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,771 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:28,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:28,774 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:28,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:28,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:28,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:28,775 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:28,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,777 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,784 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,786 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:28,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,786 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,786 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:28,786 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:28,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:28,790 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,790 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/0c5f4e219676499086e1c8760a454e45 2023-04-06 05:57:28,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2cd8e09159b04e5f8ee217c931f7a290 2023-04-06 05:57:28,887 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:28,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:28,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:28,897 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:28,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:28,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:28,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/f7155f12a9c2445fab9d2438c65ad71c 2023-04-06 05:57:28,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/0c5f4e219676499086e1c8760a454e45 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45 2023-04-06 05:57:28,906 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45, entries=200, sequenceid=275, filesize=14.3 K 2023-04-06 05:57:28,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2cd8e09159b04e5f8ee217c931f7a290 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290 2023-04-06 05:57:28,916 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,916 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:28,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:28,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:28,926 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:28,926 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:28,926 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:28,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:28,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:28,927 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,927 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:28,927 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:28,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:28,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:28,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:28,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:28,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290, entries=150, sequenceid=275, filesize=11.9 K 2023-04-06 05:57:28,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/f7155f12a9c2445fab9d2438c65ad71c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c 2023-04-06 05:57:28,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:28,938 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c, entries=150, sequenceid=275, filesize=11.9 K 2023-04-06 05:57:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for 61b6df3b1818e224b3645def0a20728d in 562ms, sequenceid=275, compaction requested=true 2023-04-06 05:57:28,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:28,948 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 05:57:28,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:28,949 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-06 05:57:28,949 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 05:57:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4212a402[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:28,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:28,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:28,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:28,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:28,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:28,963 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94084 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 05:57:28,963 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:28,963 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,963 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/41eec27da08b45e4a01db217a771998f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=91.9 K 2023-04-06 05:57:28,964 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 41eec27da08b45e4a01db217a771998f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:28,964 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c3feda578a444c13bfcbf07dfffd49b9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680760641674 2023-04-06 05:57:28,964 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5d5f7d4c455241628ef264045d058d58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760642295 2023-04-06 05:57:28,965 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting acac03d75fac4a529cfc09bf89b6ca54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680760643868 2023-04-06 05:57:28,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96524 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 05:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:28,965 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:28,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:28,966 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:28,966 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/ea770357afe942f0af7ebf99612b2d03, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=94.3 K 2023-04-06 05:57:28,966 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2ff6b9f8843d4220a716a2cc9c106039, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680760644459 2023-04-06 05:57:28,967 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:28,970 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:28,970 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:28,970 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:28,970 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:28,970 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:28,972 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 9755ff7673bb42e4b9a1a0f2dbaf2992, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680760645647 2023-04-06 05:57:28,972 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ea770357afe942f0af7ebf99612b2d03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:28,974 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3e89981ff0f3463abceb03fcebba63c8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680760641674 2023-04-06 05:57:28,974 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1ae9f105826e4981ad736d29784e1f0a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680760646807 2023-04-06 05:57:28,975 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting cc9fdfed40344f2e817152ea0cba8f37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760642295 2023-04-06 05:57:28,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,976 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2cd8e09159b04e5f8ee217c931f7a290, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:28,976 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a493720c76fa4786b0b78b4674e194a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680760643868 2023-04-06 05:57:28,976 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting cfa79df4b00f464a9fccf6f0fbab5006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680760644459 2023-04-06 05:57:28,977 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4d802145d40b459797f6b2f6f1c50410, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680760645647 2023-04-06 05:57:28,978 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e06270c1541048df86c2f37ac6a8d5da, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680760646807 2023-04-06 05:57:28,978 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 0c5f4e219676499086e1c8760a454e45, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:28,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:28,998 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:28,998 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:28,998 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,009 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#219 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:29,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:29,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:29,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:29,026 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#220 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:29,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760649046, max timestamp=1680760649059Num uniques 859; 2023-04-06 05:57:29,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:29,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760649046, max timestamp=1680760649059Num uniques 859; 2023-04-06 05:57:29,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:29,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760649046, max timestamp=1680760649059Num uniques 859; 2023-04-06 05:57:29,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:29,098 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,104 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,119 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,119 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,119 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,119 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,119 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,122 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae80463[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,124 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,125 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,125 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760709119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760709119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,129 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,129 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,129 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,130 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760709121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760709121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760709127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,134 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,137 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,137 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760709228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760709228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,238 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760709233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760709236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760709236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,250 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,253 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,253 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,258 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,258 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,265 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25270add[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,272 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,272 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,277 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,277 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:29,377 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,384 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,384 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,384 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,384 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,385 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,385 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,385 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,385 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,386 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,386 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,389 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,389 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,389 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,389 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,390 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec12624[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,398 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,401 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,401 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,401 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,402 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,405 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,405 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,405 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,405 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,405 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760709436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,441 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,441 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,441 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760709436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760709442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760709444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760709446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cb90958c81c24bca86545e8088ef5b75 2023-04-06 05:57:29,464 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/e9da896ee76f4177a4a1f9eaec354f6f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/e9da896ee76f4177a4a1f9eaec354f6f 2023-04-06 05:57:29,470 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into e9da896ee76f4177a4a1f9eaec354f6f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:29,470 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:29,470 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=8, startTime=1680760648948; duration=0sec 2023-04-06 05:57:29,470 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:29,470 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-06 05:57:29,474 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 94084 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-06 05:57:29,475 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:29,475 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,475 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/232bc8f1d0e840d898adfac869e5500d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=91.9 K 2023-04-06 05:57:29,476 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 232bc8f1d0e840d898adfac869e5500d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1680760641609 2023-04-06 05:57:29,477 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ba0bf5463fa54fc4ac1815ead016a344, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680760641674 2023-04-06 05:57:29,477 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d20db5f7e8fb469bb0e4aa75eeb8d02e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760642295 2023-04-06 05:57:29,480 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 957fe62627534dc09c6f26eec842765b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680760643868 2023-04-06 05:57:29,481 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 13ffd61d76be4c039d998b93d4404d9d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1680760644459 2023-04-06 05:57:29,482 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a1bbed09903945d7947fafe35e86fb88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680760645647 2023-04-06 05:57:29,482 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/2f24f8a41f4049beb1cd09e7b7a5328e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/2f24f8a41f4049beb1cd09e7b7a5328e 2023-04-06 05:57:29,482 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 91eb9bb125c74789b1f3848f320b7521, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680760646807 2023-04-06 05:57:29,483 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f7155f12a9c2445fab9d2438c65ad71c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:29,489 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into 2f24f8a41f4049beb1cd09e7b7a5328e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:29,490 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:29,490 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=8, startTime=1680760648948; duration=0sec 2023-04-06 05:57:29,490 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:29,506 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#223 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:29,529 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/5bf00d78be644e1ab9ced594681ea3c5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5bf00d78be644e1ab9ced594681ea3c5 2023-04-06 05:57:29,541 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,543 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into 5bf00d78be644e1ab9ced594681ea3c5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:29,543 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:29,543 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=8, startTime=1680760648949; duration=0sec 2023-04-06 05:57:29,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,543 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:29,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,559 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,560 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,580 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,580 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,582 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,583 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,583 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2924af4a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,584 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,584 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,586 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,622 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,622 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,622 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,623 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,623 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,723 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,735 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,736 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,736 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,736 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,736 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,738 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,738 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,738 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760709743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760709743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760709749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760709749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,751 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:29,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760709752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63256d11[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,760 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,761 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:29,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,766 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,766 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,866 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:29,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:29,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:29,869 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:29,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:29,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:29,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:29,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:29,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:29,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:29,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:29,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:29,886 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:29,890 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,890 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:29,890 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:29,890 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:29,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:29,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,895 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:29,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:29,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e49f24a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:29,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:29,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:29,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:29,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:29,902 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:29,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:29,908 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:29,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:29,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:29,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:29,908 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:29,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:29,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/deaa2f82b2f34db78ae1ef0565ae40a8 2023-04-06 05:57:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:29,928 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:29,930 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:29,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:29,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:29,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:29,932 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:29,932 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:29,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:29,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:30,033 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:30,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:30,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:30,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:30,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:30,046 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:30,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:30,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:30,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:30,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:30,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:30,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:30,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:30,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:30,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:30,053 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,053 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:30,053 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:30,053 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:30,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:30,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:30,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:30,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,060 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,060 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,060 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:30,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9ce0ba1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:30,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:30,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,065 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:30,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:30,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:30,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:30,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:30,067 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:30,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:30,067 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:30,068 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,068 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,068 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,069 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,070 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:30,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,072 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:30,072 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,172 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:30,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:30,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:30,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:30,176 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:30,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:30,176 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:30,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,179 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:30,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:30,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:30,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:30,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:30,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:30,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:30,187 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,187 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:30,188 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:30,188 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:30,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:30,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:30,195 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:30,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cab2d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:30,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:30,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,201 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:30,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:30,201 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:30,201 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:30,205 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:30,205 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,205 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:30,210 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:30,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:30,212 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:30,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:30,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:30,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:30,212 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:30,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,222 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:30,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:30,227 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:30,227 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:30,227 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43822 deadline: 1680760710248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760710249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43916 deadline: 1680760710254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:30,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760710258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:30,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760710265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,327 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:30,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:30,332 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:30,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:30,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:30,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:30,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:30,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:30,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:30,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:30,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:30,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:30,339 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:30,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:30,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,343 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:30,346 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:30,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d19137a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:30,348 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:30,348 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,351 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,351 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:30,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:30,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:30,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:30,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:30,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/ef1bd83786e2435fb97172200f357287 2023-04-06 05:57:30,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:30,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:30,362 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:30,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:30,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:30,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:30,362 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:30,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cb90958c81c24bca86545e8088ef5b75 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75 2023-04-06 05:57:30,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75, entries=200, sequenceid=285, filesize=14.3 K 2023-04-06 05:57:30,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/deaa2f82b2f34db78ae1ef0565ae40a8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8 2023-04-06 05:57:30,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,371 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,371 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:30,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:30,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:30,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:30,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,376 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:30,376 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:30,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:30,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:30,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8, entries=150, sequenceid=285, filesize=11.9 K 2023-04-06 05:57:30,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/ef1bd83786e2435fb97172200f357287 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287 2023-04-06 05:57:30,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287, entries=150, sequenceid=285, filesize=11.9 K 2023-04-06 05:57:30,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 61b6df3b1818e224b3645def0a20728d in 1361ms, sequenceid=285, compaction requested=false 2023-04-06 05:57:30,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:30,476 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:30,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:30,481 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:30,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:30,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:30,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,500 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:30,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:30,500 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:30,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:30,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:30,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:30,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:30,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:30,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:30,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:30,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:30,504 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:30,504 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:30,504 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:30,504 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:30,523 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cef0f4860b3a43ef956d576ee69427bf 2023-04-06 05:57:30,544 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/19b8f1c7c51046718024a47d4a313f36 2023-04-06 05:57:30,968 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/fa13905a71624431974a689fa3433f24 2023-04-06 05:57:30,971 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/cef0f4860b3a43ef956d576ee69427bf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf 2023-04-06 05:57:30,974 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf, entries=150, sequenceid=324, filesize=11.9 K 2023-04-06 05:57:30,976 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/19b8f1c7c51046718024a47d4a313f36 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36 2023-04-06 05:57:30,984 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36, entries=150, sequenceid=324, filesize=11.9 K 2023-04-06 05:57:30,985 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/fa13905a71624431974a689fa3433f24 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24 2023-04-06 05:57:30,990 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24, entries=150, sequenceid=324, filesize=11.9 K 2023-04-06 05:57:30,991 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 61b6df3b1818e224b3645def0a20728d in 487ms, sequenceid=324, compaction requested=true 2023-04-06 05:57:30,991 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:30,991 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:30,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:30,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:30,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:30,991 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:30,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:30,992 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:30,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf37331[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,012 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,013 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,016 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,016 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,018 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,021 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,021 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,121 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,130 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,130 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,130 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,130 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,135 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,135 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,135 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:31,135 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:31,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:31,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,137 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519fc0d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,138 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,138 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,147 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,148 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,167 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,167 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,176 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,267 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:31,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:31,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:31,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,270 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:31,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,270 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,270 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:31,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:31,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:31,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:31,271 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:31,273 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:31,275 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,275 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653c6a40[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760651285, max timestamp=1680760651292Num uniques 859; 2023-04-06 05:57:31,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:31,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760651285, max timestamp=1680760651292Num uniques 859; 2023-04-06 05:57:31,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:31,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760651285, max timestamp=1680760651292Num uniques 859; 2023-04-06 05:57:31,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:31,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,309 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,309 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,313 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,313 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,313 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760711312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760711313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760711314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760711314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760711315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,413 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760711419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,421 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760711422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760711422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,423 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760711422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,423 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,423 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760711423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:31,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:31,430 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:31,435 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,435 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60bb375e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,446 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,454 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,463 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,464 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,466 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,469 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:31,469 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,469 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,569 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,574 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,578 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,578 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,580 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,584 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,584 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,584 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,586 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,586 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,586 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:31,586 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:31,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:31,593 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b6ba83[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,598 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,606 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,607 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,615 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,616 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,616 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,616 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:31,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760711624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760711625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43858 deadline: 1680760711626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760711626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760711627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3c6a570c0c564c83ace2cf289e21531b 2023-04-06 05:57:31,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/ce64634dadaf4b93944a8576a9bce741 2023-04-06 05:57:31,716 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,720 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,725 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,725 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,725 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:31,725 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:31,725 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:31,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:31,736 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:31,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d05c7ec[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:31,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:31,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,739 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/857f3bcd182c4c12bcdd0a92aaef1017 2023-04-06 05:57:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:31,741 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:31,741 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,741 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:31,741 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:31,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,742 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:31,752 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:31,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:31,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:31,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:31,752 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:31,753 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,753 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,753 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:31,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:31,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:31,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:31,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:31,758 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:31,758 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:31,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:31,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:31,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3c6a570c0c564c83ace2cf289e21531b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b 2023-04-06 05:57:31,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b, entries=150, sequenceid=336, filesize=11.9 K 2023-04-06 05:57:31,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/ce64634dadaf4b93944a8576a9bce741 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741 2023-04-06 05:57:31,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741, entries=150, sequenceid=336, filesize=11.9 K 2023-04-06 05:57:31,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/857f3bcd182c4c12bcdd0a92aaef1017 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017 2023-04-06 05:57:31,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017, entries=150, sequenceid=336, filesize=11.9 K 2023-04-06 05:57:31,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 61b6df3b1818e224b3645def0a20728d in 519ms, sequenceid=336, compaction requested=true 2023-04-06 05:57:31,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:31,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:31,788 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:31,788 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:31,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:31,788 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:31,791 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,791 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/2f24f8a41f4049beb1cd09e7b7a5328e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=50.5 K 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2f24f8a41f4049beb1cd09e7b7a5328e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:31,791 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cb90958c81c24bca86545e8088ef5b75, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680760648395 2023-04-06 05:57:31,791 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,792 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/e9da896ee76f4177a4a1f9eaec354f6f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=48.1 K 2023-04-06 05:57:31,792 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cef0f4860b3a43ef956d576ee69427bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680760649117 2023-04-06 05:57:31,792 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e9da896ee76f4177a4a1f9eaec354f6f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:31,792 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3c6a570c0c564c83ace2cf289e21531b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:31,792 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting deaa2f82b2f34db78ae1ef0565ae40a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680760648400 2023-04-06 05:57:31,793 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 19b8f1c7c51046718024a47d4a313f36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680760649117 2023-04-06 05:57:31,794 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ce64634dadaf4b93944a8576a9bce741, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:31,803 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:31,812 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:31,851 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/1ece897cbb23461ab6bc63dd0ea9ba3c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ece897cbb23461ab6bc63dd0ea9ba3c 2023-04-06 05:57:31,858 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:31,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:31,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:31,862 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/98bd242e202847ab99a0300e0c906156 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/98bd242e202847ab99a0300e0c906156 2023-04-06 05:57:31,865 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into 1ece897cbb23461ab6bc63dd0ea9ba3c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:31,865 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:31,865 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=12, startTime=1680760651788; duration=0sec 2023-04-06 05:57:31,865 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:31,865 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:31,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:31,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:31,868 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:31,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:31,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:31,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:31,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:31,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:31,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:31,876 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:31,876 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:31,876 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,877 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into 98bd242e202847ab99a0300e0c906156(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:31,877 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:31,877 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=12, startTime=1680760651788; duration=0sec 2023-04-06 05:57:31,877 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:31,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:31,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:31,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:31,877 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5bf00d78be644e1ab9ced594681ea3c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=48.1 K 2023-04-06 05:57:31,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:31,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:31,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:31,885 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:31,885 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:31,885 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:31,886 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:31,886 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:31,886 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5bf00d78be644e1ab9ced594681ea3c5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680760648059 2023-04-06 05:57:31,887 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ef1bd83786e2435fb97172200f357287, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680760648400 2023-04-06 05:57:31,887 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting fa13905a71624431974a689fa3433f24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680760649117 2023-04-06 05:57:31,888 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 857f3bcd182c4c12bcdd0a92aaef1017, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:31,912 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#233 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:31,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:31,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:31,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760711948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760711948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760711948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760711949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:31,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760711957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:31,958 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/1300e748c3634f55b63c0bc8fd4ba87b 2023-04-06 05:57:32,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760712053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760712053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760712057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760712057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760712058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760712255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760712258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760712261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760712262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760712263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,339 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/368d8e4ba31a40ddb24e79ff31834bcb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/368d8e4ba31a40ddb24e79ff31834bcb 2023-04-06 05:57:32,343 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into 368d8e4ba31a40ddb24e79ff31834bcb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:32,343 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:32,343 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=12, startTime=1680760651788; duration=0sec 2023-04-06 05:57:32,344 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:32,411 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/9dcbb4c10c2242eba542c205ca429265 2023-04-06 05:57:32,433 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/3e7a4d231ef94d65aae4c651794be18c 2023-04-06 05:57:32,436 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/1300e748c3634f55b63c0bc8fd4ba87b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b 2023-04-06 05:57:32,441 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b, entries=150, sequenceid=374, filesize=11.9 K 2023-04-06 05:57:32,443 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/9dcbb4c10c2242eba542c205ca429265 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265 2023-04-06 05:57:32,448 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265, entries=150, sequenceid=374, filesize=11.9 K 2023-04-06 05:57:32,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/3e7a4d231ef94d65aae4c651794be18c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c 2023-04-06 05:57:32,466 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c, entries=150, sequenceid=374, filesize=11.9 K 2023-04-06 05:57:32,467 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for 61b6df3b1818e224b3645def0a20728d in 582ms, sequenceid=374, compaction requested=false 2023-04-06 05:57:32,467 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:32,467 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:32,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:32,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:32,470 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:32,471 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,471 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:32,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6dda66[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:32,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:32,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:32,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:32,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:32,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:32,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:32,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:32,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:32,478 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:32,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:32,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:32,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:32,478 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:32,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:32,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:32,486 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:32,486 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:32,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:32,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:32,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:57:32,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:32,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:32,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:32,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:32,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:32,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:32,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760712575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760712575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760712575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760712576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760712576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,586 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:32,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:32,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:32,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:32,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:32,591 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:32,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:32,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:32,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,592 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:32,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:32,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:32,593 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,593 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:32,593 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:32,593 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:32,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:32,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:32,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:32,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:32,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72550592[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:32,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:32,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:32,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:32,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:32,598 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:32,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:32,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:32,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:32,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:32,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:32,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:32,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,604 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:32,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:32,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,604 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,604 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:32,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760712677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760712678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760712678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760712679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760712679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,704 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:32,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:32,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:32,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:32,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:32,707 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:32,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:32,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:32,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:32,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:32,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:32,709 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:32,709 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,709 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:32,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:32,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:32,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:32,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:32,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:32,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4005dbf9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:32,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:32,714 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:32,715 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:32,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:32,717 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:32,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:32,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:32,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:32,717 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:32,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:32,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:32,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:32,722 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,722 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,722 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:32,722 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,722 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,822 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:32,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:32,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:32,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:32,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:32,824 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:32,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:32,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:32,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:32,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:32,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:32,827 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,827 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:32,827 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:32,827 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:32,827 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:32,828 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,828 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:32,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dc5eb2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:32,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:32,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,830 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:32,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,831 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:32,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:32,833 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:32,833 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:32,833 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:32,833 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:32,833 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:32,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,835 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:32,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:32,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,838 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:32,838 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:32,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760712879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760712880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760712880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760712881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:32,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760712881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,938 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:32,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:32,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:32,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:32,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:32,941 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:32,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:32,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:32,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,942 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:32,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:32,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:32,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:32,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:32,943 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:32,943 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:32,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:32,945 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:32,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c70e45[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:32,945 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,946 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:32,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:32,947 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:32,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:32,948 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:32,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:32,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:32,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:32,949 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:32,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:32,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:32,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:32,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:32,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:32,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:32,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:32,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:32,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:32,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:32,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:32,953 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:32,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:32,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:32,953 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:32,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/77f9e5e7d0014bbdb1977dfe8b63bcd6 2023-04-06 05:57:33,053 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,056 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,057 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,058 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,058 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,058 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,059 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,060 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,060 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce4f29a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,065 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,065 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,066 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,066 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:33,069 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,069 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,069 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,169 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,172 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,172 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,172 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,172 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,173 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,173 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,173 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,173 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,174 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,174 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,176 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,176 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,176 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,176 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d020ec[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,179 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,180 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,180 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760713181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760713182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,185 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,185 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760713184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760713185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760713185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,285 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,288 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,289 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,289 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,289 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,289 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,290 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,290 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,290 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,290 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7338c1ca[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,293 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,295 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,295 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,296 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,296 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,296 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,296 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,297 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:33,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,299 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,299 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/7cfcaa31fa394f59bad2926f679f6863 2023-04-06 05:57:33,399 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,402 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,403 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,403 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,403 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,404 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,404 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,404 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d334cce[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,408 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,409 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,412 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,412 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,421 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,421 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,422 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,422 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:33,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,522 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,525 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b817d0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,533 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,533 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,537 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,537 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,537 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,538 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,538 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,538 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,637 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,641 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,643 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc96ab[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,646 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,646 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:33,653 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,653 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760713685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760713687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760713688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760713688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:33,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760713689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,753 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:33,758 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:33,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:33,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:33,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:33,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30df314b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:33,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:33,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:33,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:33,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:33,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:33,764 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:33,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:33,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:33,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:33,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:33,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,767 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:33,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:33,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,770 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:33,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,770 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:33,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:33,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:33,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:33,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4145139a3e0e4b95803be66d15107487 2023-04-06 05:57:33,791 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/77f9e5e7d0014bbdb1977dfe8b63bcd6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6 2023-04-06 05:57:33,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6, entries=200, sequenceid=387, filesize=14.3 K 2023-04-06 05:57:33,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/7cfcaa31fa394f59bad2926f679f6863 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863 2023-04-06 05:57:33,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863, entries=150, sequenceid=387, filesize=11.9 K 2023-04-06 05:57:33,801 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4145139a3e0e4b95803be66d15107487 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487 2023-04-06 05:57:33,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487, entries=150, sequenceid=387, filesize=11.9 K 2023-04-06 05:57:33,806 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 61b6df3b1818e224b3645def0a20728d in 1245ms, sequenceid=387, compaction requested=true 2023-04-06 05:57:33,807 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:33,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:33,807 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:33,807 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:33,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:33,807 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:33,808 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:33,808 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:33,808 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,808 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ece897cbb23461ab6bc63dd0ea9ba3c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=36.3 K 2023-04-06 05:57:33,808 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:33,808 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:33,808 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,809 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/98bd242e202847ab99a0300e0c906156, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=38.7 K 2023-04-06 05:57:33,809 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1ece897cbb23461ab6bc63dd0ea9ba3c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:33,809 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 98bd242e202847ab99a0300e0c906156, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:33,809 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9dcbb4c10c2242eba542c205ca429265, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680760651310 2023-04-06 05:57:33,809 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1300e748c3634f55b63c0bc8fd4ba87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680760651310 2023-04-06 05:57:33,809 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7cfcaa31fa394f59bad2926f679f6863, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:33,810 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 77f9e5e7d0014bbdb1977dfe8b63bcd6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:33,819 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:33,835 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:33,870 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/0f2b992fd358404aab9609b1bb660afe as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/0f2b992fd358404aab9609b1bb660afe 2023-04-06 05:57:33,870 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:33,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:33,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:33,874 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:33,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:33,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:33,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:33,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:33,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:33,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:33,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:33,877 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into 0f2b992fd358404aab9609b1bb660afe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:33,877 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:33,877 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=13, startTime=1680760653807; duration=0sec 2023-04-06 05:57:33,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:33,877 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:33,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:33,877 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:33,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:33,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:33,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:33,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:33,878 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:33,878 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:33,878 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:57:33,878 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:33,878 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/368d8e4ba31a40ddb24e79ff31834bcb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=36.3 K 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:33,879 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 368d8e4ba31a40ddb24e79ff31834bcb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680760651263 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:33,879 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:33,879 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3e7a4d231ef94d65aae4c651794be18c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680760651310 2023-04-06 05:57:33,879 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4145139a3e0e4b95803be66d15107487, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:33,883 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/24251942dee24a868aa714bc63775198 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/24251942dee24a868aa714bc63775198 2023-04-06 05:57:33,886 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:33,888 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into 24251942dee24a868aa714bc63775198(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:33,888 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:33,888 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=13, startTime=1680760653807; duration=0sec 2023-04-06 05:57:33,888 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:33,894 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/01514a40a36f4a9199463ea0a2c736b9 2023-04-06 05:57:33,913 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/536a665f13794b1b8c0a193d1a0b373d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/536a665f13794b1b8c0a193d1a0b373d 2023-04-06 05:57:33,917 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into 536a665f13794b1b8c0a193d1a0b373d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:33,917 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:33,917 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=13, startTime=1680760653807; duration=0sec 2023-04-06 05:57:33,917 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:34,319 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/37cc2bf6163e43a49952e3168f317241 2023-04-06 05:57:34,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:34,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:34,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760714698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760714698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760714700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43858 deadline: 1680760714700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760714701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,732 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/0bb98604ea734ddeaa9966c72399b63e 2023-04-06 05:57:34,735 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/01514a40a36f4a9199463ea0a2c736b9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9 2023-04-06 05:57:34,738 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9, entries=150, sequenceid=408, filesize=11.9 K 2023-04-06 05:57:34,739 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/37cc2bf6163e43a49952e3168f317241 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241 2023-04-06 05:57:34,741 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241, entries=150, sequenceid=408, filesize=11.9 K 2023-04-06 05:57:34,742 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/0bb98604ea734ddeaa9966c72399b63e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e 2023-04-06 05:57:34,745 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e, entries=150, sequenceid=408, filesize=11.9 K 2023-04-06 05:57:34,745 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 61b6df3b1818e224b3645def0a20728d in 867ms, sequenceid=408, compaction requested=false 2023-04-06 05:57:34,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:34,746 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:34,746 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,747 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,747 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:34,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675d5ea2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:34,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:34,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:34,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:34,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:34,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:34,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:34,751 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:34,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:34,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:34,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:34,751 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:34,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,753 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,754 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:34,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:34,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,756 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:34,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,756 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:34,755 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:34,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:57:34,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:34,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:34,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:34,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:34,804 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:34,804 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760714813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760714814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760714815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760714815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,856 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:34,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:34,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:34,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:34,858 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:34,858 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:34,858 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:34,858 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:34,858 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,858 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,859 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:34,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,859 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:34,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:34,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:34,860 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:34,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:34,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:34,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:34,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:34,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:34,862 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:34,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112dd30e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:34,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:34,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,863 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:34,864 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,864 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:34,864 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,864 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:34,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:34,866 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:34,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:34,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:34,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:34,866 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:34,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:34,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,870 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:34,870 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:34,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760714916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760714917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760714917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:34,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760714918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,970 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:34,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:34,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:34,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:34,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:34,973 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:34,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:34,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:34,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:34,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,974 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:34,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:34,974 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,974 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:34,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:34,975 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:34,975 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:34,975 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:34,976 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:34,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,976 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:34,978 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a5664e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:34,978 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,978 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:34,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:34,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:34,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:34,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:34,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:34,982 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:34,982 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:34,982 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:34,982 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:34,982 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:34,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:34,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,984 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:34,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:34,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:34,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:34,986 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:34,986 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:34,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:34,986 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,086 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,089 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,089 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,089 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,089 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,089 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,089 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,091 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,093 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39395478[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,095 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,096 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,102 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,102 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760715120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760715121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760715121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760715122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,203 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,205 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/90470133e2004beb86755852af45323b 2023-04-06 05:57:35,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@349ca5b3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,215 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,215 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,221 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,221 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,321 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,324 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd6a58d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,357 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,357 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,359 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,359 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,360 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,360 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,360 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,360 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,361 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,361 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760715422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760715423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760715425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760715426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,461 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,464 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,465 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,466 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,466 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,466 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,466 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,467 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@101a2daf[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,472 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,472 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,472 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,472 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,473 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,475 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,475 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,479 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,480 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,480 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,480 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,480 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,480 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,580 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,582 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,583 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,583 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,583 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,583 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,584 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,584 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,585 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,585 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,585 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,585 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,585 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,586 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,587 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,587 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,589 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f32140[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,589 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,590 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,590 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,590 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,592 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,592 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,593 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,593 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,593 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,597 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,597 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2d3f6a9a82384bd881f7bda8d07800a2 2023-04-06 05:57:35,697 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,699 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,701 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,701 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,703 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,703 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,703 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3213a1b0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,704 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,704 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,704 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,705 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,707 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,707 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,708 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,711 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,711 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,711 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,811 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,814 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,816 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,818 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1790b2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,821 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,822 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,823 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,824 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,826 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,826 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:35,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,926 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:35,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:35,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:35,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:35,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:35,929 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:35,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:35,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:35,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43906 deadline: 1680760715927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,930 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43874 deadline: 1680760715927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43822 deadline: 1680760715929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,930 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,930 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,931 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,931 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:35,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:35,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:35,931 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:35,931 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:35,931 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:35,932 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,932 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:35,932 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:35,932 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:35,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:35,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:35,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43916 deadline: 1680760715931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:35,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da1abca[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:35,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:35,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,935 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:35,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:35,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:35,936 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:35,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:35,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:35,937 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:35,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:35,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:35,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:35,937 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:35,938 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,938 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:35,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,938 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:35,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,938 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,939 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:35,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:35,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:35,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:35,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:35,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:35,941 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:35,941 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:35,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:35,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:35,941 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:36,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4cb125634dc64a09a4f3fc524bc225f4 2023-04-06 05:57:36,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/90470133e2004beb86755852af45323b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b 2023-04-06 05:57:36,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b, entries=200, sequenceid=424, filesize=14.3 K 2023-04-06 05:57:36,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2d3f6a9a82384bd881f7bda8d07800a2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2 2023-04-06 05:57:36,041 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:36,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2, entries=150, sequenceid=424, filesize=11.9 K 2023-04-06 05:57:36,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:36,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/4cb125634dc64a09a4f3fc524bc225f4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4 2023-04-06 05:57:36,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:36,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4, entries=150, sequenceid=424, filesize=11.9 K 2023-04-06 05:57:36,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 61b6df3b1818e224b3645def0a20728d in 1242ms, sequenceid=424, compaction requested=true 2023-04-06 05:57:36,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:36,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:36,046 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:36,046 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:36,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:36,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:36,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:36,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:36,047 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:36,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:36,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:36,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:36,047 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:36,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:36,047 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/A is initiating minor compaction (all files) 2023-04-06 05:57:36,047 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:36,047 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:36,047 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/B is initiating minor compaction (all files) 2023-04-06 05:57:36,047 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/A in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:36,047 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/B in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:36,047 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/24251942dee24a868aa714bc63775198, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=38.8 K 2023-04-06 05:57:36,047 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/0f2b992fd358404aab9609b1bb660afe, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=36.4 K 2023-04-06 05:57:36,048 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 24251942dee24a868aa714bc63775198, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:36,048 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 0f2b992fd358404aab9609b1bb660afe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:36,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:36,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:36,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:36,048 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 01514a40a36f4a9199463ea0a2c736b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680760652574 2023-04-06 05:57:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:36,048 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 37cc2bf6163e43a49952e3168f317241, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680760652574 2023-04-06 05:57:36,048 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 90470133e2004beb86755852af45323b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680760654699 2023-04-06 05:57:36,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:36,049 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2d3f6a9a82384bd881f7bda8d07800a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680760654699 2023-04-06 05:57:36,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:36,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:36,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:36,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:36,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. started... 2023-04-06 05:57:36,050 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:36,050 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:36,054 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#B#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:36,055 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#A#compaction#251 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:36,066 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/3bc0e73a38be4d06bc8f1d1c96fed4c7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3bc0e73a38be4d06bc8f1d1c96fed4c7 2023-04-06 05:57:36,070 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/A of 61b6df3b1818e224b3645def0a20728d into 3bc0e73a38be4d06bc8f1d1c96fed4c7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:36,071 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:36,071 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/A, priority=13, startTime=1680760656046; duration=0sec 2023-04-06 05:57:36,071 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:36,071 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:57:36,071 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:57:36,072 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 61b6df3b1818e224b3645def0a20728d/C is initiating minor compaction (all files) 2023-04-06 05:57:36,072 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61b6df3b1818e224b3645def0a20728d/C in TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:36,072 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/536a665f13794b1b8c0a193d1a0b373d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp, totalSize=36.4 K 2023-04-06 05:57:36,072 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 536a665f13794b1b8c0a193d1a0b373d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1680760651936 2023-04-06 05:57:36,072 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0bb98604ea734ddeaa9966c72399b63e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680760652574 2023-04-06 05:57:36,072 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4cb125634dc64a09a4f3fc524bc225f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680760654699 2023-04-06 05:57:36,076 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61b6df3b1818e224b3645def0a20728d#C#compaction#252 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:36,177 DEBUG [Thread-1430] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e5e7a93 to 127.0.0.1:55165 2023-04-06 05:57:36,177 DEBUG [Thread-1428] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509adaa8 to 127.0.0.1:55165 2023-04-06 05:57:36,177 DEBUG [Thread-1428] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,177 DEBUG [Thread-1430] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,178 DEBUG [Thread-1424] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bae1a64 to 127.0.0.1:55165 2023-04-06 05:57:36,178 DEBUG [Thread-1424] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,178 DEBUG [Thread-1426] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5654f8ca to 127.0.0.1:55165 2023-04-06 05:57:36,178 DEBUG [Thread-1426] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,179 DEBUG [Thread-1432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3254ddef to 127.0.0.1:55165 2023-04-06 05:57:36,179 DEBUG [Thread-1432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,462 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/de1e4e966fd946e6b09cd82e0a3e0e62 2023-04-06 05:57:36,472 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/701793bf0fcc477dbc53513c464cc0b8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/701793bf0fcc477dbc53513c464cc0b8 2023-04-06 05:57:36,477 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/B of 61b6df3b1818e224b3645def0a20728d into 701793bf0fcc477dbc53513c464cc0b8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:36,477 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:36,477 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/B, priority=13, startTime=1680760656046; duration=0sec 2023-04-06 05:57:36,477 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:36,485 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/c3aedf9e67504bcbab6d1ad666f444b1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/c3aedf9e67504bcbab6d1ad666f444b1 2023-04-06 05:57:36,490 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61b6df3b1818e224b3645def0a20728d/C of 61b6df3b1818e224b3645def0a20728d into c3aedf9e67504bcbab6d1ad666f444b1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:36,490 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:36,490 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d., storeName=61b6df3b1818e224b3645def0a20728d/C, priority=13, startTime=1680760656046; duration=0sec 2023-04-06 05:57:36,491 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:36,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:36,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. as already flushing 2023-04-06 05:57:36,727 DEBUG [Thread-1415] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407b9cb0 to 127.0.0.1:55165 2023-04-06 05:57:36,727 DEBUG [Thread-1415] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,879 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/b4f1acbb164b42e3ac98da84d2d94985 2023-04-06 05:57:36,937 DEBUG [Thread-1413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x712d78ca to 127.0.0.1:55165 2023-04-06 05:57:36,937 DEBUG [Thread-1417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f456fa2 to 127.0.0.1:55165 2023-04-06 05:57:36,937 DEBUG [Thread-1419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62be7a0b to 127.0.0.1:55165 2023-04-06 05:57:36,937 DEBUG [Thread-1417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,937 DEBUG [Thread-1413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,937 DEBUG [Thread-1419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:36,942 DEBUG [Thread-1421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7abacd49 to 127.0.0.1:55165 2023-04-06 05:57:36,942 DEBUG [Thread-1421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:37,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/138ec5123f8b447ca4ea66da2f88ec37 2023-04-06 05:57:37,312 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/de1e4e966fd946e6b09cd82e0a3e0e62 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/de1e4e966fd946e6b09cd82e0a3e0e62 2023-04-06 05:57:37,317 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/de1e4e966fd946e6b09cd82e0a3e0e62, entries=150, sequenceid=442, filesize=11.9 K 2023-04-06 05:57:37,318 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/b4f1acbb164b42e3ac98da84d2d94985 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b4f1acbb164b42e3ac98da84d2d94985 2023-04-06 05:57:37,322 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b4f1acbb164b42e3ac98da84d2d94985, entries=150, sequenceid=442, filesize=11.9 K 2023-04-06 05:57:37,323 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/138ec5123f8b447ca4ea66da2f88ec37 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/138ec5123f8b447ca4ea66da2f88ec37 2023-04-06 05:57:37,327 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/138ec5123f8b447ca4ea66da2f88ec37, entries=150, sequenceid=442, filesize=11.9 K 2023-04-06 05:57:37,327 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=33.54 KB/34350 for 61b6df3b1818e224b3645def0a20728d in 1277ms, sequenceid=442, compaction requested=false 2023-04-06 05:57:37,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:37,327 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:37,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:37,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:37,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:37,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:37,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:37,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:37,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:37,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a2721f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:37,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:37,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:37,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:37,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:37,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:37,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:37,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:37,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:37,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:37,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:37,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:37,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:37,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:37,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:37,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:37,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:37,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:37,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:37,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:37,336 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:37,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:37,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:37,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:37,336 DEBUG [Thread-1423] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:37,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:37,336 DEBUG [Thread-1423] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:37,436 DEBUG [Thread-1423] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:37,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4429 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4482 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4453 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4429 2023-04-06 05:57:37,438 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 4483 2023-04-06 05:57:37,439 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:57:37,439 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:57:37,439 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7187fcf3 to 127.0.0.1:55165 2023-04-06 05:57:37,439 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:57:37,439 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:57:37,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:57:37,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:37,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:37,444 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760657444"}]},"ts":"1680760657444"} 2023-04-06 05:57:37,445 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:57:37,446 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:57:37,447 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, UNASSIGN}] 2023-04-06 05:57:37,449 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, UNASSIGN 2023-04-06 05:57:37,450 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=61b6df3b1818e224b3645def0a20728d, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:37,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760657450"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760657450"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760657450"}]},"ts":"1680760657450"} 2023-04-06 05:57:37,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:37,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:37,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:37,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 61b6df3b1818e224b3645def0a20728d, disabling compactions & flushes 2023-04-06 05:57:37,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:37,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:37,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. after waiting 0 ms 2023-04-06 05:57:37,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:37,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 61b6df3b1818e224b3645def0a20728d 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 05:57:37,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=A 2023-04-06 05:57:37,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:37,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=B 2023-04-06 05:57:37,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:37,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61b6df3b1818e224b3645def0a20728d, store=C 2023-04-06 05:57:37,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:38,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/5d9075bbfdd44868ac7a536b32e8ade4 2023-04-06 05:57:38,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:38,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2c48206ff45b4d1288abd628474bde0f 2023-04-06 05:57:38,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:38,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/af284aa4a0cd4146aa4ed423e2f41abf 2023-04-06 05:57:38,865 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/A/5d9075bbfdd44868ac7a536b32e8ade4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/5d9075bbfdd44868ac7a536b32e8ade4 2023-04-06 05:57:38,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/5d9075bbfdd44868ac7a536b32e8ade4, entries=150, sequenceid=453, filesize=11.9 K 2023-04-06 05:57:38,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/B/2c48206ff45b4d1288abd628474bde0f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2c48206ff45b4d1288abd628474bde0f 2023-04-06 05:57:38,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2c48206ff45b4d1288abd628474bde0f, entries=150, sequenceid=453, filesize=11.9 K 2023-04-06 05:57:38,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/.tmp/C/af284aa4a0cd4146aa4ed423e2f41abf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/af284aa4a0cd4146aa4ed423e2f41abf 2023-04-06 05:57:38,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/af284aa4a0cd4146aa4ed423e2f41abf, entries=150, sequenceid=453, filesize=11.9 K 2023-04-06 05:57:38,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 61b6df3b1818e224b3645def0a20728d in 1274ms, sequenceid=453, compaction requested=true 2023-04-06 05:57:38,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/7ae63179354944369cef4d39f7f27598, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/ea770357afe942f0af7ebf99612b2d03, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/2f24f8a41f4049beb1cd09e7b7a5328e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/98bd242e202847ab99a0300e0c906156, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/24251942dee24a868aa714bc63775198, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b] to archive 2023-04-06 05:57:38,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:38,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/348988be617c441097c5f1a6c05a345f 2023-04-06 05:57:38,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/133b77d5d09e468784b54056ac2d473e 2023-04-06 05:57:38,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/7ae63179354944369cef4d39f7f27598 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/7ae63179354944369cef4d39f7f27598 2023-04-06 05:57:38,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/19d942f384f94a80bbac365f82634bd0 2023-04-06 05:57:38,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3cb8b281a66b4a86aab92611ceb8df48 2023-04-06 05:57:38,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/07eb3436b2d343d9814865d1192c32b1 2023-04-06 05:57:38,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/ea770357afe942f0af7ebf99612b2d03 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/ea770357afe942f0af7ebf99612b2d03 2023-04-06 05:57:38,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/8d4ca8919828414a932a4e93c772ee8f 2023-04-06 05:57:38,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3e89981ff0f3463abceb03fcebba63c8 2023-04-06 05:57:38,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cc9fdfed40344f2e817152ea0cba8f37 2023-04-06 05:57:38,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/a493720c76fa4786b0b78b4674e194a7 2023-04-06 05:57:38,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cfa79df4b00f464a9fccf6f0fbab5006 2023-04-06 05:57:38,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/4d802145d40b459797f6b2f6f1c50410 2023-04-06 05:57:38,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/e06270c1541048df86c2f37ac6a8d5da 2023-04-06 05:57:38,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/0c5f4e219676499086e1c8760a454e45 2023-04-06 05:57:38,911 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/2f24f8a41f4049beb1cd09e7b7a5328e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/2f24f8a41f4049beb1cd09e7b7a5328e 2023-04-06 05:57:38,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cb90958c81c24bca86545e8088ef5b75 2023-04-06 05:57:38,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/cef0f4860b3a43ef956d576ee69427bf 2023-04-06 05:57:38,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/98bd242e202847ab99a0300e0c906156 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/98bd242e202847ab99a0300e0c906156 2023-04-06 05:57:38,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3c6a570c0c564c83ace2cf289e21531b 2023-04-06 05:57:38,916 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/1300e748c3634f55b63c0bc8fd4ba87b 2023-04-06 05:57:38,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/77f9e5e7d0014bbdb1977dfe8b63bcd6 2023-04-06 05:57:38,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/24251942dee24a868aa714bc63775198 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/24251942dee24a868aa714bc63775198 2023-04-06 05:57:38,918 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/01514a40a36f4a9199463ea0a2c736b9 2023-04-06 05:57:38,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/90470133e2004beb86755852af45323b 2023-04-06 05:57:38,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5753f9c3c6544c80bdf9279ec61e83b2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/41eec27da08b45e4a01db217a771998f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/e9da896ee76f4177a4a1f9eaec354f6f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ece897cbb23461ab6bc63dd0ea9ba3c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/0f2b992fd358404aab9609b1bb660afe, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2] to archive 2023-04-06 05:57:38,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:38,923 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/82f65370af824c33a19c31db7c0938b7 2023-04-06 05:57:38,924 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/3cc363061a664524b0410b9ce9fdacb9 2023-04-06 05:57:38,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5753f9c3c6544c80bdf9279ec61e83b2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5753f9c3c6544c80bdf9279ec61e83b2 2023-04-06 05:57:38,926 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b5f89370cb3c4cf2aa42ce85d28bbcd6 2023-04-06 05:57:38,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/514e59e07d234664ad97cb1367a57890 2023-04-06 05:57:38,928 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/372a0e38e85848c6bcd72648e8263eb8 2023-04-06 05:57:38,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/41eec27da08b45e4a01db217a771998f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/41eec27da08b45e4a01db217a771998f 2023-04-06 05:57:38,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/093a9303526b40a89e450b55802295d5 2023-04-06 05:57:38,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/c3feda578a444c13bfcbf07dfffd49b9 2023-04-06 05:57:38,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/5d5f7d4c455241628ef264045d058d58 2023-04-06 05:57:38,932 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/acac03d75fac4a529cfc09bf89b6ca54 2023-04-06 05:57:38,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2ff6b9f8843d4220a716a2cc9c106039 2023-04-06 05:57:38,933 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9755ff7673bb42e4b9a1a0f2dbaf2992 2023-04-06 05:57:38,934 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ae9f105826e4981ad736d29784e1f0a 2023-04-06 05:57:38,935 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/e9da896ee76f4177a4a1f9eaec354f6f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/e9da896ee76f4177a4a1f9eaec354f6f 2023-04-06 05:57:38,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2cd8e09159b04e5f8ee217c931f7a290 2023-04-06 05:57:38,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/deaa2f82b2f34db78ae1ef0565ae40a8 2023-04-06 05:57:38,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/19b8f1c7c51046718024a47d4a313f36 2023-04-06 05:57:38,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ece897cbb23461ab6bc63dd0ea9ba3c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/1ece897cbb23461ab6bc63dd0ea9ba3c 2023-04-06 05:57:38,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/ce64634dadaf4b93944a8576a9bce741 2023-04-06 05:57:38,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/9dcbb4c10c2242eba542c205ca429265 2023-04-06 05:57:38,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/0f2b992fd358404aab9609b1bb660afe to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/0f2b992fd358404aab9609b1bb660afe 2023-04-06 05:57:38,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/7cfcaa31fa394f59bad2926f679f6863 2023-04-06 05:57:38,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/37cc2bf6163e43a49952e3168f317241 2023-04-06 05:57:38,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2d3f6a9a82384bd881f7bda8d07800a2 2023-04-06 05:57:38,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b7e7144abc5141b68b2c5005fa4fda61, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/232bc8f1d0e840d898adfac869e5500d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5bf00d78be644e1ab9ced594681ea3c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/368d8e4ba31a40ddb24e79ff31834bcb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/536a665f13794b1b8c0a193d1a0b373d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4] to archive 2023-04-06 05:57:38,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:57:38,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a84c69ba2d874c3f9c03ea8f136ff15e 2023-04-06 05:57:38,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5852af8b4f744f968f5c8ec6d1ccf294 2023-04-06 05:57:38,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b7e7144abc5141b68b2c5005fa4fda61 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b7e7144abc5141b68b2c5005fa4fda61 2023-04-06 05:57:38,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0775a5a68ef14c5296ad89e281949e8c 2023-04-06 05:57:38,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/691c2905c39c40dc8f3578720f164f96 2023-04-06 05:57:38,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4e6ee0880e1f4ecea130824df24e6646 2023-04-06 05:57:38,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/232bc8f1d0e840d898adfac869e5500d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/232bc8f1d0e840d898adfac869e5500d 2023-04-06 05:57:38,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/b73b2c43b3b74787afe9195af29ad8af 2023-04-06 05:57:38,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ba0bf5463fa54fc4ac1815ead016a344 2023-04-06 05:57:38,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/d20db5f7e8fb469bb0e4aa75eeb8d02e 2023-04-06 05:57:38,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/957fe62627534dc09c6f26eec842765b 2023-04-06 05:57:38,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/13ffd61d76be4c039d998b93d4404d9d 2023-04-06 05:57:38,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/a1bbed09903945d7947fafe35e86fb88 2023-04-06 05:57:38,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/91eb9bb125c74789b1f3848f320b7521 2023-04-06 05:57:38,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5bf00d78be644e1ab9ced594681ea3c5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/5bf00d78be644e1ab9ced594681ea3c5 2023-04-06 05:57:38,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/f7155f12a9c2445fab9d2438c65ad71c 2023-04-06 05:57:38,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/ef1bd83786e2435fb97172200f357287 2023-04-06 05:57:38,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/fa13905a71624431974a689fa3433f24 2023-04-06 05:57:38,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/368d8e4ba31a40ddb24e79ff31834bcb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/368d8e4ba31a40ddb24e79ff31834bcb 2023-04-06 05:57:38,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/857f3bcd182c4c12bcdd0a92aaef1017 2023-04-06 05:57:38,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/3e7a4d231ef94d65aae4c651794be18c 2023-04-06 05:57:38,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/536a665f13794b1b8c0a193d1a0b373d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/536a665f13794b1b8c0a193d1a0b373d 2023-04-06 05:57:38,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4145139a3e0e4b95803be66d15107487 2023-04-06 05:57:38,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/0bb98604ea734ddeaa9966c72399b63e 2023-04-06 05:57:38,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/4cb125634dc64a09a4f3fc524bc225f4 2023-04-06 05:57:38,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/recovered.edits/456.seqid, newMaxSeqId=456, maxSeqId=1 2023-04-06 05:57:38,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d. 2023-04-06 05:57:38,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 61b6df3b1818e224b3645def0a20728d: 2023-04-06 05:57:38,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:38,973 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=61b6df3b1818e224b3645def0a20728d, regionState=CLOSED 2023-04-06 05:57:38,974 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760658973"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760658973"}]},"ts":"1680760658973"} 2023-04-06 05:57:38,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-06 05:57:38,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 61b6df3b1818e224b3645def0a20728d, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.5230 sec 2023-04-06 05:57:38,976 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-06 05:57:38,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61b6df3b1818e224b3645def0a20728d, UNASSIGN in 1.5280 sec 2023-04-06 05:57:38,977 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760658977"}]},"ts":"1680760658977"} 2023-04-06 05:57:38,978 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:57:38,979 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:57:38,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5390 sec 2023-04-06 05:57:39,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-06 05:57:39,552 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-06 05:57:39,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:57:39,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,560 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,562 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 05:57:39,569 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:39,571 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/recovered.edits] 2023-04-06 05:57:39,575 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3bc0e73a38be4d06bc8f1d1c96fed4c7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/3bc0e73a38be4d06bc8f1d1c96fed4c7 2023-04-06 05:57:39,576 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/5d9075bbfdd44868ac7a536b32e8ade4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/5d9075bbfdd44868ac7a536b32e8ade4 2023-04-06 05:57:39,577 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/de1e4e966fd946e6b09cd82e0a3e0e62 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/A/de1e4e966fd946e6b09cd82e0a3e0e62 2023-04-06 05:57:39,580 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2c48206ff45b4d1288abd628474bde0f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/2c48206ff45b4d1288abd628474bde0f 2023-04-06 05:57:39,582 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/701793bf0fcc477dbc53513c464cc0b8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/701793bf0fcc477dbc53513c464cc0b8 2023-04-06 05:57:39,583 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b4f1acbb164b42e3ac98da84d2d94985 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/B/b4f1acbb164b42e3ac98da84d2d94985 2023-04-06 05:57:39,587 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/138ec5123f8b447ca4ea66da2f88ec37 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/138ec5123f8b447ca4ea66da2f88ec37 2023-04-06 05:57:39,589 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/af284aa4a0cd4146aa4ed423e2f41abf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/af284aa4a0cd4146aa4ed423e2f41abf 2023-04-06 05:57:39,591 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/c3aedf9e67504bcbab6d1ad666f444b1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/C/c3aedf9e67504bcbab6d1ad666f444b1 2023-04-06 05:57:39,595 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/recovered.edits/456.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d/recovered.edits/456.seqid 2023-04-06 05:57:39,596 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/61b6df3b1818e224b3645def0a20728d 2023-04-06 05:57:39,596 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:57:39,599 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,601 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:57:39,603 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:57:39,605 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,605 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:57:39,605 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760659605"}]},"ts":"9223372036854775807"} 2023-04-06 05:57:39,607 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:57:39,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 61b6df3b1818e224b3645def0a20728d, NAME => 'TestAcidGuarantees,,1680760633867.61b6df3b1818e224b3645def0a20728d.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:57:39,607 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:57:39,607 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760659607"}]},"ts":"9223372036854775807"} 2023-04-06 05:57:39,609 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:57:39,612 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-04-06 05:57:39,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-06 05:57:39,665 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-06 05:57:39,681 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=545 (was 502) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=412 (was 272) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3532 (was 3568) 2023-04-06 05:57:39,691 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=412, ProcessCount=169, AvailableMemoryMB=3532 2023-04-06 05:57:39,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:57:39,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:57:39,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:39,695 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:57:39,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-06 05:57:39,696 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:57:39,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:39,697 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:39,697 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 empty. 2023-04-06 05:57:39,697 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:39,697 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:57:39,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:39,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:40,108 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:57:40,114 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => efa7c061e44c974bf8b2c1105ad04a72, NAME => 'TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:57:40,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:40,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:57:40,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing efa7c061e44c974bf8b2c1105ad04a72, disabling compactions & flushes 2023-04-06 05:57:40,524 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. after waiting 0 ms 2023-04-06 05:57:40,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,524 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,524 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:40,531 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:57:40,533 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760660533"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760660533"}]},"ts":"1680760660533"} 2023-04-06 05:57:40,534 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:57:40,536 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:57:40,536 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760660536"}]},"ts":"1680760660536"} 2023-04-06 05:57:40,537 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:57:40,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, ASSIGN}] 2023-04-06 05:57:40,543 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, ASSIGN 2023-04-06 05:57:40,544 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:57:40,695 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:40,696 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760660695"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760660695"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760660695"}]},"ts":"1680760660695"} 2023-04-06 05:57:40,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:40,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:40,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => efa7c061e44c974bf8b2c1105ad04a72, NAME => 'TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:57:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:57:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,861 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,863 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A 2023-04-06 05:57:40,864 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A 2023-04-06 05:57:40,864 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:40,864 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName A 2023-04-06 05:57:40,865 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:40,865 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,867 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B 2023-04-06 05:57:40,867 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B 2023-04-06 05:57:40,868 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:40,868 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName B 2023-04-06 05:57:40,869 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:40,869 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,871 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C 2023-04-06 05:57:40,871 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C 2023-04-06 05:57:40,871 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:40,871 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName C 2023-04-06 05:57:40,872 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:40,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:57:40,878 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:40,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:57:40,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened efa7c061e44c974bf8b2c1105ad04a72; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74917931, jitterRate=0.11636416614055634}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:57:40,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:40,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., pid=37, masterSystemTime=1680760660850 2023-04-06 05:57:40,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:40,885 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:40,885 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760660885"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760660885"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760660885"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760660885"}]},"ts":"1680760660885"} 2023-04-06 05:57:40,889 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-06 05:57:40,889 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 in 188 msec 2023-04-06 05:57:40,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-06 05:57:40,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, ASSIGN in 348 msec 2023-04-06 05:57:40,892 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:57:40,892 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760660892"}]},"ts":"1680760660892"} 2023-04-06 05:57:40,893 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:57:40,895 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:57:40,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2020 sec 2023-04-06 05:57:41,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-06 05:57:41,807 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-06 05:57:41,808 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x28a98302 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:41,818 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d63a8ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:41,820 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:41,824 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:41,825 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:57:41,826 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:57:41,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:57:41,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:57:41,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 05:57:42,245 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:57:42,248 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:57:42,248 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:57:42,253 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 05:57:42,256 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, REOPEN/MOVE}] 2023-04-06 05:57:42,257 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, REOPEN/MOVE 2023-04-06 05:57:42,258 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,258 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760662258"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760662258"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760662258"}]},"ts":"1680760662258"} 2023-04-06 05:57:42,259 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:42,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing efa7c061e44c974bf8b2c1105ad04a72, disabling compactions & flushes 2023-04-06 05:57:42,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. after waiting 0 ms 2023-04-06 05:57:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 05:57:42,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:42,417 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: efa7c061e44c974bf8b2c1105ad04a72 to self. 2023-04-06 05:57:42,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,419 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=CLOSED 2023-04-06 05:57:42,419 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760662419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760662419"}]},"ts":"1680760662419"} 2023-04-06 05:57:42,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-06 05:57:42,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 in 161 msec 2023-04-06 05:57:42,423 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=true 2023-04-06 05:57:42,574 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,574 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760662574"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760662574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760662574"}]},"ts":"1680760662574"} 2023-04-06 05:57:42,575 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:57:42,730 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => efa7c061e44c974bf8b2c1105ad04a72, NAME => 'TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:57:42,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:57:42,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,732 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,732 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A 2023-04-06 05:57:42,732 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A 2023-04-06 05:57:42,733 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:42,733 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName A 2023-04-06 05:57:42,734 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:42,735 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,736 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B 2023-04-06 05:57:42,736 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B 2023-04-06 05:57:42,736 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:42,736 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName B 2023-04-06 05:57:42,736 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:42,737 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,737 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C 2023-04-06 05:57:42,737 DEBUG [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C 2023-04-06 05:57:42,738 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:57:42,738 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efa7c061e44c974bf8b2c1105ad04a72 columnFamilyName C 2023-04-06 05:57:42,738 INFO [StoreOpener-efa7c061e44c974bf8b2c1105ad04a72-1] regionserver.HStore(310): Store=efa7c061e44c974bf8b2c1105ad04a72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:57:42,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:57:42,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened efa7c061e44c974bf8b2c1105ad04a72; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69860595, jitterRate=0.041003987193107605}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:57:42,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:42,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., pid=42, masterSystemTime=1680760662726 2023-04-06 05:57:42,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,745 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,745 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760662745"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760662745"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760662745"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760662745"}]},"ts":"1680760662745"} 2023-04-06 05:57:42,747 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-06 05:57:42,747 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 in 171 msec 2023-04-06 05:57:42,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-06 05:57:42,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, REOPEN/MOVE in 491 msec 2023-04-06 05:57:42,750 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-06 05:57:42,750 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 496 msec 2023-04-06 05:57:42,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 922 msec 2023-04-06 05:57:42,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-06 05:57:42,752 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x39d5c003 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,756 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d80e5a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,756 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x39e3a826 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,760 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ea1a2c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,760 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x019e7fff to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,785 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25185572, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,786 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x47e30d1c to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,791 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35f0f398, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,791 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x36a9ece3 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,794 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69fd9cc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,794 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x54da4907 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,798 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5144fcff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,798 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x52cea5eb to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,803 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4cc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,803 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x3546dad6 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,809 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1280c068, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,809 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x4965c362 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,813 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4af630f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,813 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x1a0b0d8a to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:57:42,816 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d12b37a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:57:42,830 DEBUG [hconnection-0x145eb991-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,830 DEBUG [hconnection-0x285588a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,830 DEBUG [hconnection-0xdff23dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,831 DEBUG [hconnection-0x4167b845-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,831 DEBUG [hconnection-0x57eceb20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,832 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,832 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,832 DEBUG [hconnection-0x7a58e243-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,837 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:42,837 DEBUG [hconnection-0x6a194815-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,839 DEBUG [hconnection-0x6eb4c728-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,839 DEBUG [hconnection-0x465ed06d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,839 DEBUG [hconnection-0xe919274-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:57:42,841 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,841 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45908, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,842 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,842 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,846 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,846 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,848 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:57:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:42,854 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:42,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:42,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:42,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:42,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:42,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:42,855 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,855 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:42,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:42,855 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,856 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:42,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:42,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:42,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:42,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:42,858 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:42,858 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:42,858 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:42,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:42,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:42,866 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:42,866 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:42,869 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 05:57:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:42,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:42,892 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:42,892 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:42,893 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:42,893 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:42,893 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:42,893 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:42,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760722910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760722911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760722914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:42,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760722916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760662908, max timestamp=1680760662910Num uniques 859; 2023-04-06 05:57:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760722916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760662908, max timestamp=1680760662910Num uniques 859; 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760662908, max timestamp=1680760662910Num uniques 859; 2023-04-06 05:57:42,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:42,926 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304067f1754fa4e014e6fbe2ad37cc845e5e2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067f1754fa4e014e6fbe2ad37cc845e5e2 2023-04-06 05:57:43,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760723032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760723138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,344 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/ee2dc6a4004042b291ede6bbbb3f2d07 2023-04-06 05:57:43,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760723345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,384 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/29f071806c7d435ea3f59589982f5626 2023-04-06 05:57:43,418 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/f66f85d7e1ea409283fbe7b7a48ecbb0 2023-04-06 05:57:43,423 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/ee2dc6a4004042b291ede6bbbb3f2d07 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07 2023-04-06 05:57:43,428 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07, entries=100, sequenceid=14, filesize=18.4 K 2023-04-06 05:57:43,430 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/29f071806c7d435ea3f59589982f5626 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626 2023-04-06 05:57:43,435 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626, entries=100, sequenceid=14, filesize=9.3 K 2023-04-06 05:57:43,436 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/f66f85d7e1ea409283fbe7b7a48ecbb0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0 2023-04-06 05:57:43,443 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0, entries=100, sequenceid=14, filesize=9.3 K 2023-04-06 05:57:43,443 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for efa7c061e44c974bf8b2c1105ad04a72 in 574ms, sequenceid=14, compaction requested=false 2023-04-06 05:57:43,444 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:43,444 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:43,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:43,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:43,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5813537a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:43,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:43,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:43,454 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:43,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:43,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:43,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,460 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:43,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:43,461 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:43,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:43,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:43,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:43,462 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:43,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:43,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:43,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:43,466 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:43,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:43,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:43,468 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:43,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:43,469 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:43,469 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:43,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:43,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:43,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:43,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:43,569 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:43,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:43,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:43,572 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:43,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:43,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:43,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:43,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:43,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:43,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:43,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:43,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:43,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:43,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:43,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:43,576 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:43,576 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:43,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:43,577 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:43,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:43,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:43,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760723651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760723967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760723968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760723971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760723971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:43,993 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ff02be66bd6a416894beb9021650964a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff02be66bd6a416894beb9021650964a 2023-04-06 05:57:44,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760724161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760724273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760724277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760724277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760724278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,403 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/0cfe32c26ec4491e9dc97765dbf366df 2023-04-06 05:57:44,427 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/a299c4abbec340b9a7f3e2509e968858 2023-04-06 05:57:44,454 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/9864b4e51efa4384810c941bf7862261 2023-04-06 05:57:44,459 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/0cfe32c26ec4491e9dc97765dbf366df as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df 2023-04-06 05:57:44,466 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df, entries=150, sequenceid=51, filesize=25.1 K 2023-04-06 05:57:44,467 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/a299c4abbec340b9a7f3e2509e968858 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858 2023-04-06 05:57:44,472 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 05:57:44,473 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/9864b4e51efa4384810c941bf7862261 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261 2023-04-06 05:57:44,477 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261, entries=150, sequenceid=51, filesize=11.6 K 2023-04-06 05:57:44,478 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for efa7c061e44c974bf8b2c1105ad04a72 in 902ms, sequenceid=51, compaction requested=false 2023-04-06 05:57:44,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:44,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:44,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:44,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:44,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:44,483 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,483 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:44,484 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,484 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f1caca[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:44,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:44,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,487 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:44,487 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:44,487 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:44,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:44,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:44,488 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:44,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:44,491 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:44,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:44,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:44,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:44,491 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:44,493 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,493 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:44,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,494 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:44,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:44,498 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:44,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:44,498 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:57:44,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:44,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:44,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:44,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:44,500 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:44,500 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:44,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:44,601 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:44,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:44,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:44,607 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:44,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:44,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:44,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:44,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:44,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:44,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:44,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:44,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:44,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:44,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:44,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:44,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:44,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:44,610 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:44,610 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:44,610 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:44,610 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:44,611 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:44,611 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:44,611 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:44,611 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:44,611 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:44,632 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406bdf8868294ae42c8b06066165978360e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bdf8868294ae42c8b06066165978360e 2023-04-06 05:57:44,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:44,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:44,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760724806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760664794, max timestamp=1680760664809Num uniques 859; 2023-04-06 05:57:44,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:44,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760664794, max timestamp=1680760664809Num uniques 859; 2023-04-06 05:57:44,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:44,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760664794, max timestamp=1680760664809Num uniques 859; 2023-04-06 05:57:44,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:44,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760724837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760724838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760724841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760724914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760724942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760724947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:44,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760724947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,068 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/cc36e62be5b04843be380a92711adef8 2023-04-06 05:57:45,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760725122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760725147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760725155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760725160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760725168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760725430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760725459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760725463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760725468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,479 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/ef4449850e164c6c94cbc175b47182c6 2023-04-06 05:57:45,508 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/4a4f4efc79824f84a8316248dcf60b5f 2023-04-06 05:57:45,518 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/cc36e62be5b04843be380a92711adef8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8 2023-04-06 05:57:45,523 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8, entries=150, sequenceid=61, filesize=25.1 K 2023-04-06 05:57:45,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/ef4449850e164c6c94cbc175b47182c6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6 2023-04-06 05:57:45,527 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 05:57:45,528 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/4a4f4efc79824f84a8316248dcf60b5f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f 2023-04-06 05:57:45,531 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 05:57:45,532 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for efa7c061e44c974bf8b2c1105ad04a72 in 922ms, sequenceid=61, compaction requested=true 2023-04-06 05:57:45,532 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:45,532 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:45,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:45,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:45,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:45,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280ecbe5[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:45,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:45,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:45,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:45,540 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:45,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:45,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:45,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:45,543 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:45,543 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:45,543 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:45,543 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:45,543 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:45,557 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,557 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:45,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,557 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:45,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,558 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:45,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:45,566 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:45,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:45,566 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:45,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:45,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:45,666 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:45,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:45,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:45,669 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:45,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:45,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:45,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:45,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:45,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:45,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:45,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:45,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:45,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:45,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:45,671 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:45,671 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:45,671 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:45,671 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:45,671 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:45,671 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:45,671 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:45,672 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:45,672 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:45,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:45,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:45,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760725982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760725983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:45,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760725991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760725992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,084 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c93555a383a74a65bcc2a2fa2cb9ccda to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c93555a383a74a65bcc2a2fa2cb9ccda 2023-04-06 05:57:46,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760726094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760726095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760726100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760726103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760726305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760726305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760726307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760726307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,501 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/345c0356af9647b29420fb29c83fcfb6 2023-04-06 05:57:46,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760726614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760726614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760726615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:46,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760726617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:46,925 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/54168f752a5446289cb997e557f35d75 2023-04-06 05:57:47,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760727123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760727124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760727125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:47,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760727130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:47,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760727187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,199 DEBUG [Thread-2030] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:57:47,349 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/343f86545458474290c64d7aa20c42a3 2023-04-06 05:57:47,353 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/345c0356af9647b29420fb29c83fcfb6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6 2023-04-06 05:57:47,360 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6, entries=150, sequenceid=97, filesize=25.1 K 2023-04-06 05:57:47,360 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/54168f752a5446289cb997e557f35d75 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75 2023-04-06 05:57:47,367 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75, entries=150, sequenceid=97, filesize=11.6 K 2023-04-06 05:57:47,368 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/343f86545458474290c64d7aa20c42a3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3 2023-04-06 05:57:47,371 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3, entries=150, sequenceid=97, filesize=11.6 K 2023-04-06 05:57:47,372 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for efa7c061e44c974bf8b2c1105ad04a72 in 1701ms, sequenceid=97, compaction requested=true 2023-04-06 05:57:47,372 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:47,372 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:47,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:47,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:47,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:47,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:47,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:47,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:47,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd87671[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:47,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:47,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,375 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:47,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:47,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:47,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:47,376 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:47,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,376 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:47,376 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:47,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:47,378 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:47,378 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:47,378 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:47,378 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:47,378 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:47,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:47,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,398 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:47,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:47,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:47,398 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:47,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,399 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,399 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:47,399 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:47,499 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:47,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:47,514 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:47,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:47,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:47,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:47,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:47,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:47,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:47,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:47,516 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:47,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:47,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:47,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:47,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:47,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:47,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:47,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:47,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:47,530 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:47,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:47,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:47,959 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040647579deb301f42e1abf711cf70c3923e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040647579deb301f42e1abf711cf70c3923e 2023-04-06 05:57:48,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:48,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:48,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760668199, max timestamp=1680760668215Num uniques 859; 2023-04-06 05:57:48,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760668199, max timestamp=1680760668215Num uniques 859; 2023-04-06 05:57:48,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:48,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760668199, max timestamp=1680760668215Num uniques 859; 2023-04-06 05:57:48,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:48,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:48,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760728248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760728249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760728250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760728258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,365 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/eca15d37bb6244feaadb501e105da83c 2023-04-06 05:57:48,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760728359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760728360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760728360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760728369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760728572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760728573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760728573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760728580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,783 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/75f1b004f67d441c90eac85b7337338e 2023-04-06 05:57:48,792 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/49032be064574e8f9671b700cab6290b 2023-04-06 05:57:48,797 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/eca15d37bb6244feaadb501e105da83c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c 2023-04-06 05:57:48,800 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c, entries=100, sequenceid=107, filesize=18.4 K 2023-04-06 05:57:48,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/75f1b004f67d441c90eac85b7337338e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e 2023-04-06 05:57:48,804 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e, entries=100, sequenceid=107, filesize=9.3 K 2023-04-06 05:57:48,804 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/49032be064574e8f9671b700cab6290b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b 2023-04-06 05:57:48,807 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b, entries=100, sequenceid=107, filesize=9.3 K 2023-04-06 05:57:48,808 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for efa7c061e44c974bf8b2c1105ad04a72 in 1278ms, sequenceid=107, compaction requested=true 2023-04-06 05:57:48,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:48,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:48,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:48,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e926413[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:48,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:48,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:48,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:48,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:48,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:48,815 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:48,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:48,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:48,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:48,815 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:48,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:48,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:48,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:48,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:48,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:48,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:48,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,819 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:48,819 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:48,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668247, max timestamp=1680760668882Num uniques 1018; 2023-04-06 05:57:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:57:48,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-06 05:57:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668247, max timestamp=1680760668882Num uniques 1018; 2023-04-06 05:57:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:57:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668247, max timestamp=1680760668882Num uniques 1018; 2023-04-06 05:57:48,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:48,885 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:48,905 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406158ac4d936f849e2a68d57c740e391cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406158ac4d936f849e2a68d57c740e391cc 2023-04-06 05:57:48,919 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:48,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:48,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:48,922 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:48,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:48,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:48,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:48,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,923 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:48,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,929 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:48,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760668916, max timestamp=1680760668928Num uniques 859; 2023-04-06 05:57:48,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:48,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760668916, max timestamp=1680760668928Num uniques 859; 2023-04-06 05:57:48,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:48,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760668916, max timestamp=1680760668928Num uniques 859; 2023-04-06 05:57:48,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:57:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:48,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:48,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:48,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:48,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:48,935 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:48,935 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:48,935 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:48,935 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:48,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:48,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,940 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:48,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:48,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28935891[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:48,942 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,942 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:48,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:48,942 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:48,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:48,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:48,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:48,946 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:48,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:48,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:48,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:48,947 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:48,948 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,948 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:48,948 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:48,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:48,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:48,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:48,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:48,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:48,953 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:48,953 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:48,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:48,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:48,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:48,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:48,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:48,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:48,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760728966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668963, max timestamp=1680760668977Num uniques 1118; 2023-04-06 05:57:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668963, max timestamp=1680760668977Num uniques 1118; 2023-04-06 05:57:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 05:57:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 05:57:48,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760668963, max timestamp=1680760668977Num uniques 1118; 2023-04-06 05:57:48,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-06 05:57:49,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680760669017, max timestamp=1680760669033Num uniques 1027; 2023-04-06 05:57:49,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680760669017, max timestamp=1680760669033Num uniques 1027; 2023-04-06 05:57:49,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-06 05:57:49,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-06 05:57:49,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680760669017, max timestamp=1680760669033Num uniques 1027; 2023-04-06 05:57:49,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-06 05:57:49,053 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,058 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,058 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,058 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,058 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,058 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,059 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,062 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,062 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,062 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,062 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,063 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,064 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,064 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769cff85[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,067 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,068 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,075 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,075 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680760669064, max timestamp=1680760669090Num uniques 1136; 2023-04-06 05:57:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-06 05:57:49,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680760669064, max timestamp=1680760669090Num uniques 1136; 2023-04-06 05:57:49,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-06 05:57:49,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680760669064, max timestamp=1680760669090Num uniques 1136; 2023-04-06 05:57:49,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-06 05:57:49,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680760669108, max timestamp=1680760669125Num uniques 1145; 2023-04-06 05:57:49,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=20380, suffix heap size=147048,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=445 2023-04-06 05:57:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680760669108, max timestamp=1680760669125Num uniques 1145; 2023-04-06 05:57:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=20380, suffix heap size=147048,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=445 2023-04-06 05:57:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=445, cellSize=20380, totalHeapSize=46520, min timestamp=1680760669108, max timestamp=1680760669125Num uniques 1145; 2023-04-06 05:57:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=20380, suffix heap size=147048,new segment heap size=46520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=445 2023-04-06 05:57:49,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760729153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760729158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760729159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760729160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,175 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,183 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,186 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,186 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,186 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,186 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,187 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,188 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79f874d9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,189 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,190 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,190 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,190 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,192 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,192 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,194 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,194 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,194 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,198 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,198 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760729262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760729263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760729268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760729268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,298 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,310 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/e94275f383944b13b2b1a1de4521d7e3 2023-04-06 05:57:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,314 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,315 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,315 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/a203dd7f2fa343af912bada72c3316bd 2023-04-06 05:57:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64d72509[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,337 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,337 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,338 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,338 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,338 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,340 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,343 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,343 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,443 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,447 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,451 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56b2aed2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,453 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,453 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,453 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,454 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,454 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,456 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,456 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,457 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,459 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,460 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,460 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760729471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760729475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760729478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760729478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,560 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,563 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,564 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,564 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,566 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,566 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,566 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,566 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c31356[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,568 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,568 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,569 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,569 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,569 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,570 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,572 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,572 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,575 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,577 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,577 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,678 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,680 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,682 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,682 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da7f9cb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,688 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,688 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,689 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,689 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,689 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,690 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,691 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,692 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,693 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,693 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,693 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,693 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,693 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/3a6376c83a2049238f2295189c80f9c3 2023-04-06 05:57:49,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/e94275f383944b13b2b1a1de4521d7e3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3 2023-04-06 05:57:49,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3, entries=250, sequenceid=146, filesize=38.9 K 2023-04-06 05:57:49,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/a203dd7f2fa343af912bada72c3316bd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd 2023-04-06 05:57:49,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 05:57:49,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/3a6376c83a2049238f2295189c80f9c3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3 2023-04-06 05:57:49,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3, entries=150, sequenceid=146, filesize=11.8 K 2023-04-06 05:57:49,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=161.02 KB/164880 for efa7c061e44c974bf8b2c1105ad04a72 in 884ms, sequenceid=146, compaction requested=true 2023-04-06 05:57:49,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:49,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:49,768 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:49,768 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:49,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:49,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:49,769 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 154714 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:49,769 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66778 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:49,769 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:57:49,769 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:57:49,770 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,770 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,770 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=65.2 K 2023-04-06 05:57:49,770 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=151.1 K 2023-04-06 05:57:49,770 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 29f071806c7d435ea3f59589982f5626, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760662856 2023-04-06 05:57:49,770 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ee2dc6a4004042b291ede6bbbb3f2d07, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760662856 2023-04-06 05:57:49,770 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a299c4abbec340b9a7f3e2509e968858, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760663023 2023-04-06 05:57:49,770 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0cfe32c26ec4491e9dc97765dbf366df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760663023 2023-04-06 05:57:49,770 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ef4449850e164c6c94cbc175b47182c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760663647 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting cc36e62be5b04843be380a92711adef8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760663647 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 54168f752a5446289cb997e557f35d75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680760664833 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 345c0356af9647b29420fb29c83fcfb6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680760664833 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 75f1b004f67d441c90eac85b7337338e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680760665944 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting eca15d37bb6244feaadb501e105da83c, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680760665944 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a203dd7f2fa343af912bada72c3316bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668255 2023-04-06 05:57:49,771 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting e94275f383944b13b2b1a1de4521d7e3, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668247 2023-04-06 05:57:49,778 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:49,783 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#277 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:49,785 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/46e66285da7147f5959063a79d01a289 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/46e66285da7147f5959063a79d01a289 2023-04-06 05:57:49,789 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into 46e66285da7147f5959063a79d01a289(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:49,790 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:49,790 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=10, startTime=1680760669768; duration=0sec 2023-04-06 05:57:49,790 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:49,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:49,790 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:49,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=167.72 KB heapSize=457.78 KB 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:49,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:49,791 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 66778 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:49,791 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:57:49,791 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,791 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=65.2 K 2023-04-06 05:57:49,792 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f66f85d7e1ea409283fbe7b7a48ecbb0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680760662856 2023-04-06 05:57:49,792 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9864b4e51efa4384810c941bf7862261, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680760663023 2023-04-06 05:57:49,793 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4a4f4efc79824f84a8316248dcf60b5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760663647 2023-04-06 05:57:49,793 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,793 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 343f86545458474290c64d7aa20c42a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680760664833 2023-04-06 05:57:49,793 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 49032be064574e8f9671b700cab6290b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680760665944 2023-04-06 05:57:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,794 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3a6376c83a2049238f2295189c80f9c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668255 2023-04-06 05:57:49,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,796 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,797 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,797 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,800 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,800 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,801 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760729790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760729791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760729792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,803 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040608aad7e6c4f74a4d8651fe8c95cdda64 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040608aad7e6c4f74a4d8651fe8c95cdda64 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,804 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,804 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2085641b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,805 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,806 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#279 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:49,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760729796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,811 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,812 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,812 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,812 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,812 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,813 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,814 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,817 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,817 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760729903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760729903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760729903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:49,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760729908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,917 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:49,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:49,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:49,920 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:49,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:49,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:49,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:49,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:49,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:49,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:49,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:49,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:49,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:49,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:49,923 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,923 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:49,923 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:49,923 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:49,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:49,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63828ae7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:49,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:49,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,927 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:49,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:49,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:49,927 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:49,929 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:49,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:49,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:49,932 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:49,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:49,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:49,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:49,932 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:49,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:49,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:49,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:49,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:49,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:49,944 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:49,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:49,944 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:49,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:49,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:49,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:49,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,045 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,048 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:50,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:50,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:50,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c847c62[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:50,054 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:50,054 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,066 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:50,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:50,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:50,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:50,070 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:50,072 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:50,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:50,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:50,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:50,072 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:50,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,086 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,087 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:50,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:50,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:50,089 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:50,089 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760730110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760730110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760730111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760730111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,189 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,195 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,195 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:50,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,202 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,202 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,202 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,203 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,203 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:50,203 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:50,212 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:50,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eafb398[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:50,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:50,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:50,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,218 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/a94b09af4c95424fb6f78416a4868bb1 2023-04-06 05:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:50,219 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:50,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:50,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:50,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:50,219 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:50,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,221 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,225 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/5ac30c86236b4c46948052067ad8a070 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/5ac30c86236b4c46948052067ad8a070 2023-04-06 05:57:50,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:50,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:50,226 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:50,227 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:50,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:50,232 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into 5ac30c86236b4c46948052067ad8a070(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:50,232 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:50,232 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=10, startTime=1680760669769; duration=0sec 2023-04-06 05:57:50,232 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:50,327 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:50,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,338 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:50,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:50,338 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:50,339 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:50,354 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,354 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,355 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:50,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:50,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7581a5ea[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:50,357 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:50,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:50,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:50,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:50,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:50,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:50,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:50,362 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:50,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:50,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:50,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:50,362 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:50,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:50,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:50,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,370 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:50,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,370 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:46024 deadline: 1680760730419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45952 deadline: 1680760730420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45992 deadline: 1680760730420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45908 deadline: 1680760730420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,470 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,474 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,474 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,475 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:50,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,478 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:50,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:50,479 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:50,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:50,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@223dc0f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:50,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:50,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:50,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:50,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:50,483 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,483 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:50,483 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,483 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:50,485 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:50,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:50,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:50,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:50,485 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:50,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:50,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:50,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:50,489 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,489 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:50,489 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,589 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,592 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,593 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,595 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,595 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d5bc745[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:50,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:50,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,599 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:50,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:50,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:50,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:50,603 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:50,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:50,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:50,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:50,603 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:50,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,604 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:50,604 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,608 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:50,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:50,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:50,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,609 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:50,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:50,609 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:50,625 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/1bebadd3e0884e16a1bb3cea766ecfb0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/1bebadd3e0884e16a1bb3cea766ecfb0 2023-04-06 05:57:50,628 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into 1bebadd3e0884e16a1bb3cea766ecfb0(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:50,628 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:50,628 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=10, startTime=1680760669768; duration=0sec 2023-04-06 05:57:50,628 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:50,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/12ecf193d5a94bb49a363a4819f84871 2023-04-06 05:57:50,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/619320cc87cd493eb2e21c266db63b8a 2023-04-06 05:57:50,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/a94b09af4c95424fb6f78416a4868bb1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1 2023-04-06 05:57:50,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1, entries=250, sequenceid=253, filesize=38.9 K 2023-04-06 05:57:50,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/12ecf193d5a94bb49a363a4819f84871 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871 2023-04-06 05:57:50,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 05:57:50,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/619320cc87cd493eb2e21c266db63b8a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a 2023-04-06 05:57:50,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a, entries=150, sequenceid=253, filesize=11.8 K 2023-04-06 05:57:50,676 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~477.66 KB/489120, currentSize=13.42 KB/13740 for efa7c061e44c974bf8b2c1105ad04a72 in 886ms, sequenceid=253, compaction requested=false 2023-04-06 05:57:50,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:50,709 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:50,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:50,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:50,715 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:50,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:50,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:50,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,717 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:50,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:50,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:50,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:50,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:50,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:50,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:50,719 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:50,719 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:50,729 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304060ca56639f4274eb0b6d018c93166bed2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ca56639f4274eb0b6d018c93166bed2 2023-04-06 05:57:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:50,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760670986, max timestamp=1680760670995Num uniques 859; 2023-04-06 05:57:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:50,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760670986, max timestamp=1680760670995Num uniques 859; 2023-04-06 05:57:50,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:50,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760670986, max timestamp=1680760670995Num uniques 859; 2023-04-06 05:57:50,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760671041, max timestamp=1680760671049Num uniques 1068; 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760671041, max timestamp=1680760671049Num uniques 1068; 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760671041, max timestamp=1680760671049Num uniques 1068; 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:57:51,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760671066, max timestamp=1680760671076Num uniques 1077; 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760671066, max timestamp=1680760671076Num uniques 1077; 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760671066, max timestamp=1680760671076Num uniques 1077; 2023-04-06 05:57:51,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:57:51,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680760671095, max timestamp=1680760671106Num uniques 1086; 2023-04-06 05:57:51,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-06 05:57:51,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680760671095, max timestamp=1680760671106Num uniques 1086; 2023-04-06 05:57:51,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-06 05:57:51,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680760671095, max timestamp=1680760671106Num uniques 1086; 2023-04-06 05:57:51,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-06 05:57:51,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760671133, max timestamp=1680760671142Num uniques 1195; 2023-04-06 05:57:51,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-06 05:57:51,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760671133, max timestamp=1680760671142Num uniques 1195; 2023-04-06 05:57:51,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760671133, max timestamp=1680760671142Num uniques 1095; 2023-04-06 05:57:51,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-04-06 05:57:51,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-06 05:57:51,156 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/aa7feb4a394f4eeb86dd7b2f927dc25f 2023-04-06 05:57:51,177 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/88570f5c9b0b4766a82632d15c4ed1b9 2023-04-06 05:57:51,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760671169, max timestamp=1680760671183Num uniques 1054; 2023-04-06 05:57:51,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 05:57:51,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760671169, max timestamp=1680760671183Num uniques 1054; 2023-04-06 05:57:51,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 05:57:51,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680760671169, max timestamp=1680760671183Num uniques 1054; 2023-04-06 05:57:51,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760671206, max timestamp=1680760671220Num uniques 1013; 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760671206, max timestamp=1680760671220Num uniques 1013; 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760671206, max timestamp=1680760671220Num uniques 1013; 2023-04-06 05:57:51,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-06 05:57:51,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680760671232, max timestamp=1680760671255Num uniques 1072; 2023-04-06 05:57:51,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680760671232, max timestamp=1680760671255Num uniques 1072; 2023-04-06 05:57:51,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1680760671232, max timestamp=1680760671255Num uniques 1072; 2023-04-06 05:57:51,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-06 05:57:51,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-06 05:57:51,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-06 05:57:51,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680760671274, max timestamp=1680760671288Num uniques 1131; 2023-04-06 05:57:51,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-04-06 05:57:51,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680760671274, max timestamp=1680760671288Num uniques 1131; 2023-04-06 05:57:51,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=15156, suffix heap size=145592,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=331 2023-04-06 05:57:51,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680760671284, max timestamp=1680760671288Num uniques 1131; 2023-04-06 05:57:51,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760671303, max timestamp=1680760671329Num uniques 1190; 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760671303, max timestamp=1680760671329Num uniques 1190; 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760671303, max timestamp=1680760671329Num uniques 1090; 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-06 05:57:51,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-06 05:57:51,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680760671341, max timestamp=1680760671353Num uniques 1149; 2023-04-06 05:57:51,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-06 05:57:51,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680760671341, max timestamp=1680760671353Num uniques 1149; 2023-04-06 05:57:51,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-06 05:57:51,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680760671341, max timestamp=1680760671353Num uniques 1149; 2023-04-06 05:57:51,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-06 05:57:51,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680760671377, max timestamp=1680760671391Num uniques 1158; 2023-04-06 05:57:51,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680760671377, max timestamp=1680760671391Num uniques 1158; 2023-04-06 05:57:51,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-06 05:57:51,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680760671377, max timestamp=1680760671391Num uniques 1158; 2023-04-06 05:57:51,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-06 05:57:51,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680760671399, max timestamp=1680760671412Num uniques 1067; 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680760671399, max timestamp=1680760671412Num uniques 1067; 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=367, cellSize=16802, totalHeapSize=38408, min timestamp=1680760671399, max timestamp=1680760671412Num uniques 1067; 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-04-06 05:57:51,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=16802, suffix heap size=138936,new segment heap size=38408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=367 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680760671429, max timestamp=1680760671443Num uniques 1226; 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=17216, suffix heap size=155472,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=376 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680760671429, max timestamp=1680760671443Num uniques 1226; 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=17216, suffix heap size=155472,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=376 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680760671429, max timestamp=1680760671443Num uniques 1226; 2023-04-06 05:57:51,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=56146, new segment data size=17216, suffix heap size=155472,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1226, new segment cells count=376 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680760671456, max timestamp=1680760671471Num uniques 1235; 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=15340, suffix heap size=156408,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=335 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680760671456, max timestamp=1680760671471Num uniques 1235; 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=15340, suffix heap size=156408,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=335 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680760671456, max timestamp=1680760671471Num uniques 1235; 2023-04-06 05:57:51,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=56560, new segment data size=15340, suffix heap size=156408,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1235, new segment cells count=335 2023-04-06 05:57:51,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680760671489, max timestamp=1680760671491Num uniques 1194; 2023-04-06 05:57:51,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-06 05:57:51,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680760671489, max timestamp=1680760671491Num uniques 1194; 2023-04-06 05:57:51,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-06 05:57:51,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680760671489, max timestamp=1680760671491Num uniques 1194; 2023-04-06 05:57:51,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-06 05:57:51,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680760671505, max timestamp=1680760671539Num uniques 1103; 2023-04-06 05:57:51,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680760671505, max timestamp=1680760671539Num uniques 1103; 2023-04-06 05:57:51,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1680760671505, max timestamp=1680760671539Num uniques 1103; 2023-04-06 05:57:51,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-06 05:57:51,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-06 05:57:51,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-06 05:57:51,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680760671555, max timestamp=1680760671565Num uniques 1112; 2023-04-06 05:57:51,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=16572, suffix heap size=143616,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=362 2023-04-06 05:57:51,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680760671555, max timestamp=1680760671565Num uniques 1112; 2023-04-06 05:57:51,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=16572, suffix heap size=143616,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=362 2023-04-06 05:57:51,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680760671555, max timestamp=1680760671565Num uniques 1112; 2023-04-06 05:57:51,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=16572, suffix heap size=143616,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=362 2023-04-06 05:57:51,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680760671575, max timestamp=1680760671590Num uniques 1221; 2023-04-06 05:57:51,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=12406, suffix heap size=154952,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=271 2023-04-06 05:57:51,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680760671575, max timestamp=1680760671590Num uniques 1221; 2023-04-06 05:57:51,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=12406, suffix heap size=154952,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=271 2023-04-06 05:57:51,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1680760671575, max timestamp=1680760671590Num uniques 1221; 2023-04-06 05:57:51,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=12406, suffix heap size=154952,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=271 2023-04-06 05:57:51,596 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ebac708faa464a5fbb8e68f8056db887 2023-04-06 05:57:51,601 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/aa7feb4a394f4eeb86dd7b2f927dc25f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f 2023-04-06 05:57:51,604 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f, entries=50, sequenceid=260, filesize=11.7 K 2023-04-06 05:57:51,605 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/88570f5c9b0b4766a82632d15c4ed1b9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9 2023-04-06 05:57:51,612 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9, entries=50, sequenceid=260, filesize=7.1 K 2023-04-06 05:57:51,612 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ebac708faa464a5fbb8e68f8056db887 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887 2023-04-06 05:57:51,617 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887, entries=50, sequenceid=260, filesize=7.1 K 2023-04-06 05:57:51,619 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=120.76 KB/123660 for efa7c061e44c974bf8b2c1105ad04a72 in 899ms, sequenceid=260, compaction requested=true 2023-04-06 05:57:51,619 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:51,619 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:51,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:51,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:51,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:51,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:51,622 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:51,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=127.47 KB heapSize=354.56 KB 2023-04-06 05:57:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:51,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e6b9405[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:51,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:51,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,636 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:51,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:51,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:51,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:51,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:51,637 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:51,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:51,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:51,642 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:51,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:51,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:51,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:51,642 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:51,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760731640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:51,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:51,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:51,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,653 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:51,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,653 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760731643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760731650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760731651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760731652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,753 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:51,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:51,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760731753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:51,756 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:51,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760731755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:51,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:51,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:51,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:51,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:51,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:51,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:51,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:51,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:51,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:51,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:51,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:51,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@663ddcd7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:51,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:51,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:51,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:51,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:51,763 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760731760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760731760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760731761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:51,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:51,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:51,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:51,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:51,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:51,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:51,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,767 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:51,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:51,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:51,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:51,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:51,770 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:51,770 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:51,870 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:51,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:51,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:51,873 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:51,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:51,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:51,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:51,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:51,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:51,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:51,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:51,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:51,877 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:51,877 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:51,877 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:51,877 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:51,878 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:51,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a62b7bb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:51,880 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:51,880 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,881 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:51,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:51,881 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:51,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:51,885 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:51,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:51,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:51,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:51,885 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:51,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:51,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:51,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,889 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:51,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:51,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:51,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:51,900 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,900 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:51,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:51,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760731958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760731959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760731965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760731965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:51,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760731966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,000 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,004 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,011 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,011 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,011 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,011 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3940265d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,024 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,025 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,027 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,027 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,033 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,033 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,053 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304063e9ab3b2f77d4e1584418946f985e778 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063e9ab3b2f77d4e1584418946f985e778 2023-04-06 05:57:52,133 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,136 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,137 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,139 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4946cc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,141 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,142 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,144 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,145 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,149 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,149 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,249 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,251 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,253 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,253 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,253 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,253 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,254 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731af616[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,256 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,259 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,259 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,260 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,261 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,263 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,264 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,264 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760732263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760732264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760732270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760732270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760732271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,364 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,367 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,367 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,369 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,369 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,369 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,369 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,369 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,370 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2a58a1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,372 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,375 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,375 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,379 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,379 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,458 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=604, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/d46344aa572944f592dc95ec633f5892 2023-04-06 05:57:52,479 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,482 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,483 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,484 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,484 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,484 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,485 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,485 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,486 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24e0e7b7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,489 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,489 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,489 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,489 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,489 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,489 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,490 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,490 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,490 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,490 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,491 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,491 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,493 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,494 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,495 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,495 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,496 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,496 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,596 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,599 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,599 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,599 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,601 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,602 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,603 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,603 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15aeaac1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,605 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,606 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,608 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,608 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,608 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,609 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,609 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,613 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,613 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,613 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,613 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,713 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,716 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,716 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,717 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,719 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,719 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31205424[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,723 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,723 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,724 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,725 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,726 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,726 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,729 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,729 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760732772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760732772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760732774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760732776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:52,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760732781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,840 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,843 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,845 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,845 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,845 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,845 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,846 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f81fdf1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,847 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,848 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,848 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,848 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,850 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,850 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,851 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,852 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,854 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:52,854 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,854 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,854 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/3b49566cf4ca4417b8b44b9082ad0775 2023-04-06 05:57:52,954 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:52,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:52,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:52,959 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:52,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:52,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:52,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:52,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:52,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:52,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:52,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:52,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:52,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:52,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:52,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:52,974 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,974 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,975 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:52,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cff60e9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:52,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:52,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,977 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:52,978 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,978 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:52,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,979 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:52,979 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:52,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:52,981 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:52,981 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:52,981 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:52,981 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:52,981 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:52,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:52,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,984 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:52,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:52,988 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:52,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:52,988 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:52,988 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:52,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:53,089 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:53,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:53,091 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:53,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,092 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:53,092 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:53,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,093 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:53,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:53,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:53,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:53,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:53,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:53,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:53,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:53,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:53,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:53,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:53,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:53,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,109 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:53,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:53,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ce73115[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:53,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:53,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:53,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:53,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:53,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:53,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:53,115 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:53,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:53,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:53,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:53,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:53,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:53,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:53,119 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,119 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,119 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,119 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,126 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:53,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:53,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,126 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:53,126 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:53,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:53,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:53,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:53,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:53,226 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:53,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:53,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:53,229 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:53,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:53,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:53,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,230 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:53,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:53,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:53,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:53,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:53,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:53,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:53,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:53,232 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,232 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:53,232 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,232 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:53,233 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:53,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd5d3d9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:53,233 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,234 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:53,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:53,234 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:53,235 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:53,235 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:53,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:53,236 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:53,236 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:53,236 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:53,236 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:53,236 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:53,237 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,237 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,237 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:53,237 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:53,238 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:53,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:53,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:53,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:53,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,239 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:53,239 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:53,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:53,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:53,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/aa66afc734f6476b98e5f09ac7576068 2023-04-06 05:57:53,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/d46344aa572944f592dc95ec633f5892 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892 2023-04-06 05:57:53,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892, entries=200, sequenceid=604, filesize=32.3 K 2023-04-06 05:57:53,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/3b49566cf4ca4417b8b44b9082ad0775 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775 2023-04-06 05:57:53,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:57:53,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/aa66afc734f6476b98e5f09ac7576068 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068 2023-04-06 05:57:53,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:57:53,290 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.44 KB/383424, currentSize=46.96 KB/48090 for efa7c061e44c974bf8b2c1105ad04a72 in 1668ms, sequenceid=604, compaction requested=true 2023-04-06 05:57:53,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:53,290 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:53,290 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:53,290 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:53,290 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:53,290 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:53,291 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:53,291 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111024 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:53,291 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:57:53,291 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:57:53,291 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,291 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,292 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/46e66285da7147f5959063a79d01a289, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=42.7 K 2023-04-06 05:57:53,292 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/1bebadd3e0884e16a1bb3cea766ecfb0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=108.4 K 2023-04-06 05:57:53,292 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 46e66285da7147f5959063a79d01a289, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668255 2023-04-06 05:57:53,292 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1bebadd3e0884e16a1bb3cea766ecfb0, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668255 2023-04-06 05:57:53,292 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 12ecf193d5a94bb49a363a4819f84871, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680760669159 2023-04-06 05:57:53,292 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a94b09af4c95424fb6f78416a4868bb1, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680760669150 2023-04-06 05:57:53,292 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 88570f5c9b0b4766a82632d15c4ed1b9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760669795 2023-04-06 05:57:53,293 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting aa7feb4a394f4eeb86dd7b2f927dc25f, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760669795 2023-04-06 05:57:53,293 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3b49566cf4ca4417b8b44b9082ad0775, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:53,293 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d46344aa572944f592dc95ec633f5892, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:53,299 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:53,304 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#289 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:53,318 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/7dd4d003da274b52b7d39cdf93fecf2b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7dd4d003da274b52b7d39cdf93fecf2b 2023-04-06 05:57:53,323 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into 7dd4d003da274b52b7d39cdf93fecf2b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:53,323 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:53,323 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=12, startTime=1680760673290; duration=0sec 2023-04-06 05:57:53,323 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:53,323 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:57:53,324 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:57:53,324 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:57:53,324 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,324 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/5ac30c86236b4c46948052067ad8a070, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=42.7 K 2023-04-06 05:57:53,324 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5ac30c86236b4c46948052067ad8a070, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680760668255 2023-04-06 05:57:53,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 619320cc87cd493eb2e21c266db63b8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1680760669159 2023-04-06 05:57:53,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ebac708faa464a5fbb8e68f8056db887, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680760669795 2023-04-06 05:57:53,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting aa66afc734f6476b98e5f09ac7576068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:53,330 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:53,339 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:53,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:53,342 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:53,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:53,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,343 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:53,343 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,344 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:53,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:53,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:53,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:53,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:53,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:53,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:53,345 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:53,346 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:53,346 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:53,346 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:53,729 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/741f10253ad5472a9f32dbab3ecb3f72 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/741f10253ad5472a9f32dbab3ecb3f72 2023-04-06 05:57:53,734 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into 741f10253ad5472a9f32dbab3ecb3f72(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:53,734 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:53,734 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=12, startTime=1680760673290; duration=0sec 2023-04-06 05:57:53,734 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:53,739 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/64f8689f60ab4269b0b16c82db58daf7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/64f8689f60ab4269b0b16c82db58daf7 2023-04-06 05:57:53,744 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into 64f8689f60ab4269b0b16c82db58daf7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:53,744 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:53,744 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=12, startTime=1680760673290; duration=0sec 2023-04-06 05:57:53,744 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:53,763 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406bc60513ab0994e7f9d0ce043660c2ad4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bc60513ab0994e7f9d0ce043660c2ad4 2023-04-06 05:57:53,770 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=615, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/181c63fe4c13494abe8cb7549c189169 2023-04-06 05:57:53,780 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/c637ed1bc5434e1680df913456f1340e 2023-04-06 05:57:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:53,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760673797, max timestamp=1680760673809Num uniques 859; 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760673797, max timestamp=1680760673809Num uniques 859; 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760673797, max timestamp=1680760673809Num uniques 859; 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:53,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:53,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760733828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760733828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760733829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760733830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760733832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760733933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760733936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760733936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760733936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:53,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760733936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760734139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760734141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760734141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760734141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760734142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,196 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/661227a4bebd405fae339694efe2125b 2023-04-06 05:57:54,200 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/181c63fe4c13494abe8cb7549c189169 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169 2023-04-06 05:57:54,203 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169, entries=150, sequenceid=615, filesize=25.4 K 2023-04-06 05:57:54,203 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/c637ed1bc5434e1680df913456f1340e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e 2023-04-06 05:57:54,209 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e, entries=150, sequenceid=615, filesize=11.9 K 2023-04-06 05:57:54,210 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/661227a4bebd405fae339694efe2125b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b 2023-04-06 05:57:54,216 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b, entries=150, sequenceid=615, filesize=11.9 K 2023-04-06 05:57:54,216 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for efa7c061e44c974bf8b2c1105ad04a72 in 870ms, sequenceid=615, compaction requested=false 2023-04-06 05:57:54,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:54,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:54,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:54,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f4bc9a7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:54,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:54,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:54,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:54,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:54,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:54,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:54,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:54,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:54,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:54,224 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:54,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:54,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:54,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:54,224 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:54,225 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:54,225 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:54,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:54,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:54,227 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:54,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:54,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:54,228 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:54,228 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:54,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:54,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:54,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:54,228 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:54,229 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,229 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,229 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:54,229 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,328 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:54,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:54,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:54,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:54,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:54,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:54,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:54,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:54,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:54,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:54,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:54,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:54,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:54,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:54,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:54,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:54,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:54,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:54,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:54,336 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:54,336 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:54,337 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:54,337 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:54,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:54,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:54,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760734450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760734450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760734451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760734453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760734454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760734555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760734556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760734557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760734557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760734560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,763 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406f6e3804c2df94b56ad94c0f19860e6f9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6e3804c2df94b56ad94c0f19860e6f9 2023-04-06 05:57:54,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760734761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760734762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760734762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760734762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:54,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760734763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760735068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760735068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760735068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760735068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760735068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,174 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=651, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/589f397f428c450c9afeb0e7a3275b49 2023-04-06 05:57:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760735577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760735578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,583 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/2d85bfd146c9479ca086000b163e85c4 2023-04-06 05:57:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760735578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760735578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760735580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:55,994 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/9078183c2d854dd2bc6d1fd129f6debd 2023-04-06 05:57:55,998 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/589f397f428c450c9afeb0e7a3275b49 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49 2023-04-06 05:57:56,007 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49, entries=150, sequenceid=651, filesize=25.4 K 2023-04-06 05:57:56,008 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/2d85bfd146c9479ca086000b163e85c4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4 2023-04-06 05:57:56,012 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4, entries=150, sequenceid=651, filesize=11.9 K 2023-04-06 05:57:56,013 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/9078183c2d854dd2bc6d1fd129f6debd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd 2023-04-06 05:57:56,024 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd, entries=150, sequenceid=651, filesize=11.9 K 2023-04-06 05:57:56,025 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for efa7c061e44c974bf8b2c1105ad04a72 in 1689ms, sequenceid=651, compaction requested=true 2023-04-06 05:57:56,025 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:56,025 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:56,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:56,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:56,025 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:56,026 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:56,027 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:56,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:56,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:56,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5814068f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:56,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:56,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:56,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:56,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:56,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:56,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:56,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:56,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:56,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:56,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:56,037 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:56,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:56,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:56,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:56,037 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:56,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:56,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:56,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,043 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:56,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:56,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:56,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:56,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:56,043 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:56,043 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:56,143 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:56,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:56,148 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:56,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:56,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:56,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:56,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:56,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:56,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:56,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:56,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,153 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:56,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:56,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:56,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:56,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:56,161 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:56,161 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:56,162 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:56,162 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:56,162 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:56,163 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:56,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:56,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:56,601 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406f6b20c43551c4f7db5196423d74f28b5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6b20c43551c4f7db5196423d74f28b5 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760676603, max timestamp=1680760676618Num uniques 859; 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760676603, max timestamp=1680760676618Num uniques 859; 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680760676603, max timestamp=1680760676618Num uniques 859; 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:56,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-06 05:57:56,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760736652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760736652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760736667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760736668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760736668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760736769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760736770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760736779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760736779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760736786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760736975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760736976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760736986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:56,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:56,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760736987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760736994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,006 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=661, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/7f7a92dc4bcc4702b238db86c34d1e59 2023-04-06 05:57:57,037 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/77a7242dd0cc47e8a7d337a441a48cbb 2023-04-06 05:57:57,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760737285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760737285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760737293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760737296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760737307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,469 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=661 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/103b8f87990e4af69854c576a531ac7a 2023-04-06 05:57:57,476 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/7f7a92dc4bcc4702b238db86c34d1e59 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59 2023-04-06 05:57:57,480 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59, entries=150, sequenceid=661, filesize=25.4 K 2023-04-06 05:57:57,481 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/77a7242dd0cc47e8a7d337a441a48cbb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb 2023-04-06 05:57:57,485 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb, entries=150, sequenceid=661, filesize=11.9 K 2023-04-06 05:57:57,486 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/103b8f87990e4af69854c576a531ac7a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a 2023-04-06 05:57:57,491 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a, entries=150, sequenceid=661, filesize=11.9 K 2023-04-06 05:57:57,492 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for efa7c061e44c974bf8b2c1105ad04a72 in 1330ms, sequenceid=661, compaction requested=true 2023-04-06 05:57:57,492 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:57,492 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:57,492 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:57,492 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:57,492 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,492 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:57,492 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:57,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,493 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:57,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:57,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:57,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dfefb01[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:57,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:57,495 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,496 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:57,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:57,497 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:57,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:57,499 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:57,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:57,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:57,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:57,499 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:57,500 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:57,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,501 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,501 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:57,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:57,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:57,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:57,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:57,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:57,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,504 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:57,504 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:57,504 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,504 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:57,604 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:57,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:57,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:57,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:57,610 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:57,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:57,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:57,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:57,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:57,611 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:57,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:57,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:57,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:57,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:57,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:57,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:57,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:57,620 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:57,620 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:57,620 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:57,621 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:57,621 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:57,650 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b06122c555014793ad51cec1f38ef9fd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b06122c555014793ad51cec1f38ef9fd 2023-04-06 05:57:57,667 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=696, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/7637ca5d7f6c4a218f4eb3525c3f195e 2023-04-06 05:57:57,693 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/beb841cbf4ff425bb3fe4e3c8b18e428 2023-04-06 05:57:57,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:57,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:57,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760737813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760737814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760737818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760737821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760737821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760737922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760737923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760737936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:57,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:57,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:45964 deadline: 1680760737936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,112 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ddefc88cac2449b887d71b7708bb08ce 2023-04-06 05:57:58,115 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/7637ca5d7f6c4a218f4eb3525c3f195e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e 2023-04-06 05:57:58,118 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e, entries=150, sequenceid=696, filesize=25.4 K 2023-04-06 05:57:58,119 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/beb841cbf4ff425bb3fe4e3c8b18e428 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428 2023-04-06 05:57:58,132 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428, entries=150, sequenceid=696, filesize=11.9 K 2023-04-06 05:57:58,133 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ddefc88cac2449b887d71b7708bb08ce as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce 2023-04-06 05:57:58,144 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce, entries=150, sequenceid=696, filesize=11.9 K 2023-04-06 05:57:58,144 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=40.25 KB/41220 for efa7c061e44c974bf8b2c1105ad04a72 in 524ms, sequenceid=696, compaction requested=true 2023-04-06 05:57:58,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:58,145 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:58,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:58,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:58,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:58,146 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:58,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:58,149 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:58,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:57:58,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c14b07c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:58,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:58,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:58,151 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:58,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:58,152 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:58,152 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,152 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:58,152 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:58,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:58,156 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:58,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:58,156 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:58,161 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,161 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,165 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,169 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:58,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,169 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:58,169 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:58,170 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,175 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406540751482aeb4a69a86ee721d378cad3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406540751482aeb4a69a86ee721d378cad3 2023-04-06 05:57:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760678205, max timestamp=1680760678218Num uniques 859; 2023-04-06 05:57:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760678205, max timestamp=1680760678218Num uniques 859; 2023-04-06 05:57:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:58,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:58,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760678205, max timestamp=1680760678218Num uniques 859; 2023-04-06 05:57:58,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:57:58,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760738248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760738260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,270 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:58,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:58,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:58,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760738261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:58,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:58,275 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:58,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760738262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,276 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:58,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:58,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:58,279 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:58,279 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,279 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:58,282 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,282 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:58,282 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:58,282 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:58,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:58,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:58,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db348a8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:58,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:58,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:58,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:58,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:58,296 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:58,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:58,299 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:58,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:58,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:58,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:58,299 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:58,300 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,300 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,300 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,301 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,312 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,313 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:58,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,313 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:58,313 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:58,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760738363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760738363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760738373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760738376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,413 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:58,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:58,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:58,458 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:58,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:58,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:58,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,459 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:58,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:58,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:58,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:58,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:58,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:58,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:58,468 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,468 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:58,469 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:58,469 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:58,469 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:58,474 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,475 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256b0def[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:58,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:58,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,477 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:58,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:58,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:58,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:58,478 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:58,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:58,480 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:58,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:58,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:58,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:58,480 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:58,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,486 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:58,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:58,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,487 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:58,487 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:58,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760738571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760738571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,587 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:58,590 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=707, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/d19300e80bb74cfdb92c9de2b79b47ae 2023-04-06 05:57:58,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:58,595 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:58,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:58,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:58,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760738586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760738586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:58,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:58,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:58,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,602 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:58,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:58,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:58,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:58,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:58,606 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:58,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:58,606 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:58,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:58,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,612 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f67dcb0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:58,614 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:58,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/f7ac1c670cdf4956975ca36df76431a1 2023-04-06 05:57:58,619 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:58,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:58,619 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:58,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:58,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:58,622 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,622 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:58,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:58,624 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:58,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:58,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:58,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:58,624 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:58,625 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,625 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,626 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,627 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:58,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:58,629 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:58,629 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:58,629 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:58,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:58,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/a40dc9292a534236a366b26845ee7736 2023-04-06 05:57:58,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/d19300e80bb74cfdb92c9de2b79b47ae as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae 2023-04-06 05:57:58,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae, entries=250, sequenceid=707, filesize=39.2 K 2023-04-06 05:57:58,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/f7ac1c670cdf4956975ca36df76431a1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1 2023-04-06 05:57:58,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1, entries=150, sequenceid=707, filesize=11.9 K 2023-04-06 05:57:58,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/a40dc9292a534236a366b26845ee7736 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736 2023-04-06 05:57:58,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736, entries=150, sequenceid=707, filesize=11.9 K 2023-04-06 05:57:58,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for efa7c061e44c974bf8b2c1105ad04a72 in 555ms, sequenceid=707, compaction requested=true 2023-04-06 05:57:58,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:58,704 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:58,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:58,705 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:58,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:58,706 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73458 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:58,707 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:57:58,707 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,707 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7dd4d003da274b52b7d39cdf93fecf2b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=71.7 K 2023-04-06 05:57:58,707 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7dd4d003da274b52b7d39cdf93fecf2b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:58,710 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170702 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:58,710 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:57:58,710 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,710 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/741f10253ad5472a9f32dbab3ecb3f72, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=166.7 K 2023-04-06 05:57:58,714 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c637ed1bc5434e1680df913456f1340e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680760671638 2023-04-06 05:57:58,716 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 741f10253ad5472a9f32dbab3ecb3f72, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:58,717 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2d85bfd146c9479ca086000b163e85c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680760673820 2023-04-06 05:57:58,718 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 77a7242dd0cc47e8a7d337a441a48cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680760674450 2023-04-06 05:57:58,718 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 181c63fe4c13494abe8cb7549c189169, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680760671638 2023-04-06 05:57:58,718 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting beb841cbf4ff425bb3fe4e3c8b18e428, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680760676650 2023-04-06 05:57:58,718 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 589f397f428c450c9afeb0e7a3275b49, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680760673820 2023-04-06 05:57:58,718 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f7ac1c670cdf4956975ca36df76431a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:57:58,719 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 7f7a92dc4bcc4702b238db86c34d1e59, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680760674450 2023-04-06 05:57:58,719 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 7637ca5d7f6c4a218f4eb3525c3f195e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680760676650 2023-04-06 05:57:58,720 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d19300e80bb74cfdb92c9de2b79b47ae, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:57:58,729 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:58,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:58,765 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#307 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:58,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:58,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:58,774 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:58,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:58,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:58,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:58,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,777 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,777 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:58,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:58,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:58,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:58,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:58,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:58,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:58,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:58,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:58,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:58,811 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:58,811 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:58,811 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:58,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:57:58,814 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#306 average throughput is 0.35 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:58,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:58,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:58,866 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c551fc00dc294858a8d39d0420dbb1e3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c551fc00dc294858a8d39d0420dbb1e3 2023-04-06 05:57:58,885 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=743, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/bf946866fce5406b976081e11ac6dc85 2023-04-06 05:57:58,901 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/b421664c555b4d5b89b1b447dd3b24c0 2023-04-06 05:57:58,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760738892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760738892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760738901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760738901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:58,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:58,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760738903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760739004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760739004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760739005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760739213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760739216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760739216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,229 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/081378ab002d4d8aa2fb70ed83f32b93 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/081378ab002d4d8aa2fb70ed83f32b93 2023-04-06 05:57:59,236 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into 081378ab002d4d8aa2fb70ed83f32b93(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:59,236 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:59,236 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=10, startTime=1680760678705; duration=0sec 2023-04-06 05:57:59,236 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:57:59,236 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-06 05:57:59,239 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73458 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-06 05:57:59,239 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:57:59,239 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,239 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/64f8689f60ab4269b0b16c82db58daf7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=71.7 K 2023-04-06 05:57:59,239 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 64f8689f60ab4269b0b16c82db58daf7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760671606 2023-04-06 05:57:59,240 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 661227a4bebd405fae339694efe2125b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1680760671638 2023-04-06 05:57:59,240 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9078183c2d854dd2bc6d1fd129f6debd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680760673820 2023-04-06 05:57:59,241 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 103b8f87990e4af69854c576a531ac7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=661, earliestPutTs=1680760674450 2023-04-06 05:57:59,241 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ddefc88cac2449b887d71b7708bb08ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=696, earliestPutTs=1680760676650 2023-04-06 05:57:59,241 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a40dc9292a534236a366b26845ee7736, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:57:59,249 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#311 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:57:59,275 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/f0741c71cfc140b8862e846415214395 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f0741c71cfc140b8862e846415214395 2023-04-06 05:57:59,278 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/a9b98b8eb090420bba653052c707c038 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a9b98b8eb090420bba653052c707c038 2023-04-06 05:57:59,281 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into f0741c71cfc140b8862e846415214395(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:59,281 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into a9b98b8eb090420bba653052c707c038(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:57:59,281 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:59,281 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:59,281 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=10, startTime=1680760678706; duration=0sec 2023-04-06 05:57:59,281 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=10, startTime=1680760678704; duration=0sec 2023-04-06 05:57:59,281 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:59,281 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:57:59,344 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=743 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/75f82f89bd144c5b935d0052af50b32f 2023-04-06 05:57:59,348 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/bf946866fce5406b976081e11ac6dc85 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85 2023-04-06 05:57:59,355 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85, entries=150, sequenceid=743, filesize=25.4 K 2023-04-06 05:57:59,356 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/b421664c555b4d5b89b1b447dd3b24c0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0 2023-04-06 05:57:59,360 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0, entries=150, sequenceid=743, filesize=11.9 K 2023-04-06 05:57:59,363 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/75f82f89bd144c5b935d0052af50b32f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f 2023-04-06 05:57:59,367 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f, entries=150, sequenceid=743, filesize=11.9 K 2023-04-06 05:57:59,368 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for efa7c061e44c974bf8b2c1105ad04a72 in 557ms, sequenceid=743, compaction requested=false 2023-04-06 05:57:59,368 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:57:59,368 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,368 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,368 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,368 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,368 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,368 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,370 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,371 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56291a70[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,373 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,374 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,374 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:59,377 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:59,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:59,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:59,377 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:59,377 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:59,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,377 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,378 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:59,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:59,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,382 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:59,382 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:59,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:57:59,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:57:59,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:57:59,430 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406455f55b81e65432ba89d8fe1797c325a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406455f55b81e65432ba89d8fe1797c325a 2023-04-06 05:57:59,482 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:59,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:59,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:59,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:59,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:59,489 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:59,490 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,490 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:59,490 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:59,490 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,490 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760739489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,491 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:59,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:59,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:59,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:59,497 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,497 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:59,497 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:59,497 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760739491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1198fe15[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,504 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,504 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,508 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,508 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:59,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,514 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:59,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:59,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:59,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:59,514 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:59,514 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,514 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,516 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:59,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:59,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,520 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:59,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,520 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:59,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,520 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:59,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760739527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760739528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760739528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760739591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760739600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,620 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:59,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:59,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:59,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:59,622 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:59,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:59,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:59,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,624 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:59,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:59,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:59,625 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:59,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,628 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,629 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,629 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce19dbc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,630 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,630 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,631 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,631 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:59,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:59,635 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:59,635 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:59,635 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:59,635 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:59,635 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:59,636 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,636 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,636 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:59,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:59,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,641 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:59,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,641 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,741 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:59,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:59,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:59,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:59,744 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:59,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:59,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:59,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,744 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,746 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,746 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:59,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:59,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:59,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:59,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:59,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:59,748 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,748 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,749 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d525ecc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,751 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:59,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:59,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:59,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:59,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:59,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:59,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:59,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:59,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:59,759 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:57:59,760 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:59,760 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760739798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:57:59,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760739806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,846 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=758, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/698f6ca715aa4b078cd512a060c8ff71 2023-04-06 05:57:59,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/7e3a4e7c15af48b3b47dcdf869d56b73 2023-04-06 05:57:59,860 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:59,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:59,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:59,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:59,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:59,870 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:59,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:59,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:59,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:59,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:59,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:59,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:59,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:59,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4336258c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,877 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,877 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,878 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,879 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:57:59,881 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:57:59,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:57:59,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:57:59,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:57:59,881 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:57:59,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,884 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,886 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:57:59,887 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:57:59,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:57:59,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:57:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:57:59,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,888 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:57:59,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,888 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:57:59,988 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:57:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:57:59,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:57:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:57:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:57:59,991 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:57:59,991 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,991 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:57:59,991 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:57:59,992 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,993 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:57:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:57:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:57:59,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:57:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:57:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:57:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:57:59,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:57:59,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:57:59,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:57:59,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:57:59,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:57:59,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:57:59,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:57:59,994 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:57:59,994 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:57:59,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:57:59,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:57:59,995 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:57:59,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:57:59,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:57:59,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:57:59,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150813a9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:57:59,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:57:59,997 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:57:59,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,998 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:57:59,997 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:57:59,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:57:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:57:59,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:57:59,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:57:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:57:59,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:00,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:00,001 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:00,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:00,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:00,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:00,001 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:00,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:00,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:00,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:00,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,006 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:00,009 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:00,010 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:00,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760740037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760740037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760740038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,110 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:00,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760740106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:00,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:00,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:00,112 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:00,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:00,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:00,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,114 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,114 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:00,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:00,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:00,118 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,118 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:00,118 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:00,118 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:00,118 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:00,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760740112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,127 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:00,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:00,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e460c9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:00,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:00,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:00,134 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:00,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:00,138 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:00,138 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:00,138 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:00,138 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:00,138 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:00,139 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,139 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,139 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,144 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:00,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:00,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:00,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,144 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:00,144 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:00,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,244 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:00,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:00,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:00,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:00,247 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:00,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:00,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:00,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,250 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:00,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:00,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:00,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:00,257 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,257 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:00,257 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:00,257 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:00,257 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:00,266 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,266 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,267 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:00,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a0fe487[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:00,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:00,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:00,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:00,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:00,271 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:00,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:00,272 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:00,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:00,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:00,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:00,272 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:00,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/84cc8e7a7c784e9dbab954c1d0292875 2023-04-06 05:58:00,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,273 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:00,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:00,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:00,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:00,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:00,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:00,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,277 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:00,277 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:00,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:00,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/698f6ca715aa4b078cd512a060c8ff71 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71 2023-04-06 05:58:00,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:00,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71, entries=200, sequenceid=758, filesize=32.3 K 2023-04-06 05:58:00,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/7e3a4e7c15af48b3b47dcdf869d56b73 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73 2023-04-06 05:58:00,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73, entries=150, sequenceid=758, filesize=11.9 K 2023-04-06 05:58:00,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/84cc8e7a7c784e9dbab954c1d0292875 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875 2023-04-06 05:58:00,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875, entries=150, sequenceid=758, filesize=11.9 K 2023-04-06 05:58:00,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for efa7c061e44c974bf8b2c1105ad04a72 in 886ms, sequenceid=758, compaction requested=true 2023-04-06 05:58:00,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:00,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:00,294 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:00,294 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:00,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:00,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:00,295 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:00,295 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:58:00,295 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85724 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:00,296 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,296 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:58:00,296 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/081378ab002d4d8aa2fb70ed83f32b93, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.2 K 2023-04-06 05:58:00,296 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,296 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a9b98b8eb090420bba653052c707c038, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=83.7 K 2023-04-06 05:58:00,296 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 081378ab002d4d8aa2fb70ed83f32b93, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:58:00,296 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a9b98b8eb090420bba653052c707c038, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:58:00,296 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b421664c555b4d5b89b1b447dd3b24c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680760678258 2023-04-06 05:58:00,296 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting bf946866fce5406b976081e11ac6dc85, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680760678258 2023-04-06 05:58:00,297 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7e3a4e7c15af48b3b47dcdf869d56b73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:00,297 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 698f6ca715aa4b078cd512a060c8ff71, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:00,308 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:00,318 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#315 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:00,325 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/a187d25bc3954d5d9727c1ed4e2346ef as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a187d25bc3954d5d9727c1ed4e2346ef 2023-04-06 05:58:00,328 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into a187d25bc3954d5d9727c1ed4e2346ef(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:00,328 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:00,328 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=13, startTime=1680760680294; duration=0sec 2023-04-06 05:58:00,328 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:00,328 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:00,329 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:00,329 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:58:00,329 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,329 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f0741c71cfc140b8862e846415214395, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.2 K 2023-04-06 05:58:00,329 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f0741c71cfc140b8862e846415214395, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=707, earliestPutTs=1680760677813 2023-04-06 05:58:00,330 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 75f82f89bd144c5b935d0052af50b32f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=743, earliestPutTs=1680760678258 2023-04-06 05:58:00,330 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 84cc8e7a7c784e9dbab954c1d0292875, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:00,335 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:00,354 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/6700088764c64128bf6912ad318822b3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/6700088764c64128bf6912ad318822b3 2023-04-06 05:58:00,358 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into 6700088764c64128bf6912ad318822b3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:00,358 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:00,358 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=13, startTime=1680760680295; duration=0sec 2023-04-06 05:58:00,358 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:00,377 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:00,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:00,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:00,380 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:00,380 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,380 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:00,380 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:00,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,381 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:00,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:00,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:00,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:00,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:00,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:00,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:00,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:00,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:00,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:00,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:00,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:00,386 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:00,386 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:00,386 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:00,386 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:58:00,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:00,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:00,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:00,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760740630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760740632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760740736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760740736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,803 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406380e8c819da7420cb95d46b46ea861a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406380e8c819da7420cb95d46b46ea861a1 2023-04-06 05:58:00,806 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=780, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/73700cd0d0154060ab6eb43beb8d5287 2023-04-06 05:58:00,813 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/630270e3001a4d88aea35f45a06f6c46 2023-04-06 05:58:00,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760740939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:00,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:00,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760740940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760741048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760741049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760741052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,221 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=780 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/19974a573a1144df9c1e891b2288350d 2023-04-06 05:58:01,224 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/73700cd0d0154060ab6eb43beb8d5287 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287 2023-04-06 05:58:01,227 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287, entries=150, sequenceid=780, filesize=25.4 K 2023-04-06 05:58:01,228 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/630270e3001a4d88aea35f45a06f6c46 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46 2023-04-06 05:58:01,231 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46, entries=150, sequenceid=780, filesize=11.9 K 2023-04-06 05:58:01,237 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/19974a573a1144df9c1e891b2288350d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d 2023-04-06 05:58:01,240 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d, entries=150, sequenceid=780, filesize=11.9 K 2023-04-06 05:58:01,241 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for efa7c061e44c974bf8b2c1105ad04a72 in 855ms, sequenceid=780, compaction requested=false 2023-04-06 05:58:01,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:01,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,246 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e7ac0f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,246 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:01,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:58:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:01,253 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,261 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,264 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,264 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,264 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,265 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,277 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,278 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,278 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760741302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760741305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,378 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,381 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,381 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,383 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,383 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,383 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,383 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,384 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,386 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d276be1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,386 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,388 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,390 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,390 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,392 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,392 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,392 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,393 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,396 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,396 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760741409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760741409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,496 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,502 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,503 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,510 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,511 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,512 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,512 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b13a2a4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,513 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,514 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,515 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,515 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,515 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,515 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,515 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,516 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,516 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,520 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,525 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,526 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,526 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,561 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/f951355890584ccfb3c0eb4b0fd75ee0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/f951355890584ccfb3c0eb4b0fd75ee0 2023-04-06 05:58:01,565 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into f951355890584ccfb3c0eb4b0fd75ee0(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-06 05:58:01,565 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:01,565 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=13, startTime=1680760680294; duration=1sec 2023-04-06 05:58:01,566 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:01,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760741613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760741613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,627 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,631 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,631 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,632 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,633 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,633 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@efd4c9d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,639 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,641 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,641 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,646 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,646 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,646 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:01,677 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ea7e92944c904eb799924d6d4c1e98cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ea7e92944c904eb799924d6d4c1e98cc 2023-04-06 05:58:01,689 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=794, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/09e749f14c724a6e8633dbe4253e0ac9 2023-04-06 05:58:01,746 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,749 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,750 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,754 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b0f1e99[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,756 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,761 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,761 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,767 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,770 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,770 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,770 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,870 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,873 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,876 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,876 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:01,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,877 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:01,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:01,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d510a7a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:01,878 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,879 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:01,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:01,879 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,879 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:01,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:01,880 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:01,882 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:01,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:01,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:01,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:01,882 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:01,883 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,883 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:01,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,883 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,886 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:01,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:01,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:01,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:01,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:01,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:01,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:01,892 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:01,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:01,892 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760741920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760741920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,992 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:01,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:01,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:01,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:01,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:01,995 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:01,995 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,995 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:01,995 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:01,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,996 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:01,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:01,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:01,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:01,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:01,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:01,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:01,999 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,999 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:01,999 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:01,999 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:01,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:02,000 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,000 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:02,001 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,001 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:02,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:02,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0185f6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:02,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:02,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,003 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:02,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:02,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:02,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:02,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:02,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:02,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:02,010 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:02,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:02,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:02,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:02,010 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:02,016 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,016 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,016 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,017 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,018 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:02,027 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,027 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:02,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:02,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:02,027 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:02,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:02,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:02,027 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:02,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:02,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:02,028 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:02,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:02,028 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:02,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:02,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:02,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/f43bd6716ee54f9185f17b4b0b03b38b 2023-04-06 05:58:02,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/1ad6bd742e944c5bb57b9918fe5bc0a7 2023-04-06 05:58:02,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/09e749f14c724a6e8633dbe4253e0ac9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9 2023-04-06 05:58:02,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9, entries=250, sequenceid=794, filesize=39.2 K 2023-04-06 05:58:02,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/f43bd6716ee54f9185f17b4b0b03b38b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b 2023-04-06 05:58:02,126 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b, entries=150, sequenceid=794, filesize=11.9 K 2023-04-06 05:58:02,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/1ad6bd742e944c5bb57b9918fe5bc0a7 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7 2023-04-06 05:58:02,128 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:02,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:02,130 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7, entries=150, sequenceid=794, filesize=11.9 K 2023-04-06 05:58:02,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for efa7c061e44c974bf8b2c1105ad04a72 in 883ms, sequenceid=794, compaction requested=true 2023-04-06 05:58:02,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:02,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:02,131 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:02,131 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:02,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:02,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:02,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:02,132 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:02,132 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:02,132 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:58:02,132 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:58:02,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:02,132 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:02,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:02,132 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:02,132 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a187d25bc3954d5d9727c1ed4e2346ef, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.3 K 2023-04-06 05:58:02,132 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/f951355890584ccfb3c0eb4b0fd75ee0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=90.7 K 2023-04-06 05:58:02,132 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:02,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:02,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:02,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:02,133 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a187d25bc3954d5d9727c1ed4e2346ef, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:02,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:02,133 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f951355890584ccfb3c0eb4b0fd75ee0, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:02,133 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 630270e3001a4d88aea35f45a06f6c46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680760679449 2023-04-06 05:58:02,133 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 73700cd0d0154060ab6eb43beb8d5287, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680760679449 2023-04-06 05:58:02,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:02,133 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f43bd6716ee54f9185f17b4b0b03b38b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:02,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:02,134 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 09e749f14c724a6e8633dbe4253e0ac9, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:02,134 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:02,134 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,134 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:02,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:02,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:02,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:02,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:02,136 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:02,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:02,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:02,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:02,138 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. started... 2023-04-06 05:58:02,138 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:58:02,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:02,141 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:02,155 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#324 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:02,187 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/444901c869c343f68f38e3f4df0675fd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/444901c869c343f68f38e3f4df0675fd 2023-04-06 05:58:02,190 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into 444901c869c343f68f38e3f4df0675fd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:02,190 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:02,190 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=13, startTime=1680760682131; duration=0sec 2023-04-06 05:58:02,190 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:02,190 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:02,191 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:02,191 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:58:02,191 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:02,191 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/6700088764c64128bf6912ad318822b3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.3 K 2023-04-06 05:58:02,191 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 6700088764c64128bf6912ad318822b3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1680760678872 2023-04-06 05:58:02,192 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 19974a573a1144df9c1e891b2288350d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=780, earliestPutTs=1680760679449 2023-04-06 05:58:02,192 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1ad6bd742e944c5bb57b9918fe5bc0a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:02,198 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:02,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:02,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. as already flushing 2023-04-06 05:58:02,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760742469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760742470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760742572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760742576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,592 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406979e681c0f35455d8d78df2814642b7b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406979e681c0f35455d8d78df2814642b7b 2023-04-06 05:58:02,608 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=813, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/c2d41981d8a944b6b119388d41490961 2023-04-06 05:58:02,612 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/af5b04fdd0204c6699f12b28a7b0d36b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/af5b04fdd0204c6699f12b28a7b0d36b 2023-04-06 05:58:02,619 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into af5b04fdd0204c6699f12b28a7b0d36b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:02,619 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:02,619 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=13, startTime=1680760682131; duration=0sec 2023-04-06 05:58:02,619 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:02,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760742777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760742783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:02,826 DEBUG [Thread-2037] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54da4907 to 127.0.0.1:55165 2023-04-06 05:58:02,826 DEBUG [Thread-2037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:02,828 DEBUG [Thread-2045] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a0b0d8a to 127.0.0.1:55165 2023-04-06 05:58:02,828 DEBUG [Thread-2045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:02,829 DEBUG [Thread-2041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3546dad6 to 127.0.0.1:55165 2023-04-06 05:58:02,829 DEBUG [Thread-2041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:02,833 DEBUG [Thread-2043] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4965c362 to 127.0.0.1:55165 2023-04-06 05:58:02,833 DEBUG [Thread-2043] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:02,837 DEBUG [Thread-2039] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52cea5eb to 127.0.0.1:55165 2023-04-06 05:58:02,837 DEBUG [Thread-2039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:03,018 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/bb492c4ff8e043689430373983779279 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bb492c4ff8e043689430373983779279 2023-04-06 05:58:03,022 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into bb492c4ff8e043689430373983779279(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:03,022 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:03,022 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=13, startTime=1680760682131; duration=0sec 2023-04-06 05:58:03,022 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:03,032 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/49766e8bacb1452fa4922b0648687b63 2023-04-06 05:58:03,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:03,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45952 deadline: 1680760743061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,062 DEBUG [Thread-2034] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:03,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:03,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:46024 deadline: 1680760743063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45964 deadline: 1680760743063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,064 DEBUG [Thread-2030] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:03,064 DEBUG [Thread-2028] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:03,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:03,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45992 deadline: 1680760743083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:03,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:45908 deadline: 1680760743088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,440 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/878dc1bb0a3d4bcfb4c7a8e510c09215 2023-04-06 05:58:03,443 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/c2d41981d8a944b6b119388d41490961 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961 2023-04-06 05:58:03,446 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961, entries=150, sequenceid=813, filesize=25.4 K 2023-04-06 05:58:03,447 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/49766e8bacb1452fa4922b0648687b63 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63 2023-04-06 05:58:03,450 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63, entries=150, sequenceid=813, filesize=11.9 K 2023-04-06 05:58:03,450 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/878dc1bb0a3d4bcfb4c7a8e510c09215 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215 2023-04-06 05:58:03,454 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215, entries=150, sequenceid=813, filesize=11.9 K 2023-04-06 05:58:03,454 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for efa7c061e44c974bf8b2c1105ad04a72 in 1316ms, sequenceid=813, compaction requested=false 2023-04-06 05:58:03,454 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:03,454 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:03,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:03,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:03,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:03,455 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:03,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:03,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,456 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:03,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7cb9af[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:03,457 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:03,457 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:03,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:03,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:03,459 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:03,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:03,460 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:03,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:03,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:03,460 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:03,460 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:03,461 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,461 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:03,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,461 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:03,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:03,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:03,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:03,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:03,464 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:03,464 DEBUG [Thread-2036] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:03,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:03,464 DEBUG [Thread-2036] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:03,464 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:03,564 DEBUG [Thread-2036] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:03,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:03,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:03,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 05:58:03,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:58:03,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:03,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:58:03,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:03,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:58:03,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:03,591 DEBUG [Thread-2026] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39d5c003 to 127.0.0.1:55165 2023-04-06 05:58:03,591 DEBUG [Thread-2026] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:03,595 DEBUG [Thread-2032] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47e30d1c to 127.0.0.1:55165 2023-04-06 05:58:03,595 DEBUG [Thread-2032] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:04,002 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069ca915a5862e4ae28b08cf88d88652cf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069ca915a5862e4ae28b08cf88d88652cf 2023-04-06 05:58:04,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=831, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/6a3d02249ff547ecb50f175d2351d04f 2023-04-06 05:58:04,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/5e638afc54d5480b9fa874525cc2d28e 2023-04-06 05:58:05,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/b10338b1bff84444902d74dc78de413f 2023-04-06 05:58:05,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/6a3d02249ff547ecb50f175d2351d04f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f 2023-04-06 05:58:05,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f, entries=150, sequenceid=831, filesize=25.4 K 2023-04-06 05:58:05,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/5e638afc54d5480b9fa874525cc2d28e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e 2023-04-06 05:58:05,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e, entries=150, sequenceid=831, filesize=11.9 K 2023-04-06 05:58:05,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/b10338b1bff84444902d74dc78de413f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f 2023-04-06 05:58:05,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f, entries=150, sequenceid=831, filesize=11.9 K 2023-04-06 05:58:05,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=6.71 KB/6870 for efa7c061e44c974bf8b2c1105ad04a72 in 1663ms, sequenceid=831, compaction requested=true 2023-04-06 05:58:05,253 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:05,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:05,254 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:05,254 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:05,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:05,254 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78888 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/A is initiating minor compaction (all files) 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/B is initiating minor compaction (all files) 2023-04-06 05:58:05,255 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/A in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:05,255 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/B in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:05,255 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bb492c4ff8e043689430373983779279, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=77.0 K 2023-04-06 05:58:05,255 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/444901c869c343f68f38e3f4df0675fd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.4 K 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting bb492c4ff8e043689430373983779279, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 444901c869c343f68f38e3f4df0675fd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c2d41981d8a944b6b119388d41490961, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680760681298 2023-04-06 05:58:05,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 49766e8bacb1452fa4922b0648687b63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680760681298 2023-04-06 05:58:05,256 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 6a3d02249ff547ecb50f175d2351d04f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680760682456 2023-04-06 05:58:05,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5e638afc54d5480b9fa874525cc2d28e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680760682456 2023-04-06 05:58:05,263 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#B#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:05,264 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#A#compaction#334 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:05,672 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/bbff6d8407e9433785b9efae36fee5c2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/bbff6d8407e9433785b9efae36fee5c2 2023-04-06 05:58:05,676 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/B of efa7c061e44c974bf8b2c1105ad04a72 into bbff6d8407e9433785b9efae36fee5c2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:05,676 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:05,677 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/B, priority=13, startTime=1680760685254; duration=0sec 2023-04-06 05:58:05,677 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:05,677 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:05,678 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:05,678 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): efa7c061e44c974bf8b2c1105ad04a72/C is initiating minor compaction (all files) 2023-04-06 05:58:05,678 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of efa7c061e44c974bf8b2c1105ad04a72/C in TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:05,678 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/af5b04fdd0204c6699f12b28a7b0d36b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp, totalSize=36.4 K 2023-04-06 05:58:05,678 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting af5b04fdd0204c6699f12b28a7b0d36b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=794, earliestPutTs=1680760680626 2023-04-06 05:58:05,679 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 878dc1bb0a3d4bcfb4c7a8e510c09215, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1680760681298 2023-04-06 05:58:05,679 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b10338b1bff84444902d74dc78de413f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680760682456 2023-04-06 05:58:05,686 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): efa7c061e44c974bf8b2c1105ad04a72#C#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 57.69 MB/second 2023-04-06 05:58:05,695 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/0e0547e21f1049e491013ff9536e6ce3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/0e0547e21f1049e491013ff9536e6ce3 2023-04-06 05:58:05,698 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/C of efa7c061e44c974bf8b2c1105ad04a72 into 0e0547e21f1049e491013ff9536e6ce3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:05,699 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:05,699 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/C, priority=13, startTime=1680760685254; duration=0sec 2023-04-06 05:58:05,699 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:06,083 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/4d98ec184d6e43e1a64a5a71eca57dca as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/4d98ec184d6e43e1a64a5a71eca57dca 2023-04-06 05:58:06,088 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in efa7c061e44c974bf8b2c1105ad04a72/A of efa7c061e44c974bf8b2c1105ad04a72 into 4d98ec184d6e43e1a64a5a71eca57dca(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:06,088 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:06,088 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72., storeName=efa7c061e44c974bf8b2c1105ad04a72/A, priority=13, startTime=1680760685254; duration=0sec 2023-04-06 05:58:06,088 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:07,099 DEBUG [Thread-2028] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39e3a826 to 127.0.0.1:55165 2023-04-06 05:58:07,099 DEBUG [Thread-2034] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a9ece3 to 127.0.0.1:55165 2023-04-06 05:58:07,099 DEBUG [Thread-2030] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x019e7fff to 127.0.0.1:55165 2023-04-06 05:58:07,099 DEBUG [Thread-2034] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:07,099 DEBUG [Thread-2028] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:07,099 DEBUG [Thread-2030] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:07,099 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:58:07,099 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 172 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 186 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 158 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 1589 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 4763 rows 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 1594 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 4781 rows 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 1586 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 4758 rows 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 1604 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 4808 rows 2023-04-06 05:58:07,100 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 1597 2023-04-06 05:58:07,101 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 4790 rows 2023-04-06 05:58:07,101 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:58:07,101 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28a98302 to 127.0.0.1:55165 2023-04-06 05:58:07,101 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:07,102 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:58:07,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:58:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:07,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:07,106 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760687106"}]},"ts":"1680760687106"} 2023-04-06 05:58:07,107 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:58:07,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:07,302 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:58:07,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, UNASSIGN}] 2023-04-06 05:58:07,304 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, UNASSIGN 2023-04-06 05:58:07,304 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:07,304 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760687304"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760687304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760687304"}]},"ts":"1680760687304"} 2023-04-06 05:58:07,306 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:07,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:07,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing efa7c061e44c974bf8b2c1105ad04a72, disabling compactions & flushes 2023-04-06 05:58:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:07,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:07,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. after waiting 0 ms 2023-04-06 05:58:07,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing efa7c061e44c974bf8b2c1105ad04a72 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 05:58:07,458 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=A 2023-04-06 05:58:07,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:07,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=B 2023-04-06 05:58:07,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:07,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK efa7c061e44c974bf8b2c1105ad04a72, store=C 2023-04-06 05:58:07,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:07,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:07,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304064e0bd4a61d8d4bd4ad5cdbcb23737ee7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e0bd4a61d8d4bd4ad5cdbcb23737ee7 2023-04-06 05:58:08,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:08,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=841, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/596195cd9d684e62afc50d08aa66c59b 2023-04-06 05:58:08,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/0c615580b3b84b909bfb6a066d61ff7b 2023-04-06 05:58:08,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ec75e2f0e2eb427d8c736fe7f0254ce4 2023-04-06 05:58:08,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/A/596195cd9d684e62afc50d08aa66c59b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/596195cd9d684e62afc50d08aa66c59b 2023-04-06 05:58:08,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/596195cd9d684e62afc50d08aa66c59b, entries=50, sequenceid=841, filesize=11.7 K 2023-04-06 05:58:08,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/B/0c615580b3b84b909bfb6a066d61ff7b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/0c615580b3b84b909bfb6a066d61ff7b 2023-04-06 05:58:08,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/0c615580b3b84b909bfb6a066d61ff7b, entries=50, sequenceid=841, filesize=7.1 K 2023-04-06 05:58:08,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/.tmp/C/ec75e2f0e2eb427d8c736fe7f0254ce4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ec75e2f0e2eb427d8c736fe7f0254ce4 2023-04-06 05:58:08,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ec75e2f0e2eb427d8c736fe7f0254ce4, entries=50, sequenceid=841, filesize=7.1 K 2023-04-06 05:58:08,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for efa7c061e44c974bf8b2c1105ad04a72 in 1262ms, sequenceid=841, compaction requested=false 2023-04-06 05:58:08,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/1bebadd3e0884e16a1bb3cea766ecfb0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/741f10253ad5472a9f32dbab3ecb3f72, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a9b98b8eb090420bba653052c707c038, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/f951355890584ccfb3c0eb4b0fd75ee0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bb492c4ff8e043689430373983779279, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f] to archive 2023-04-06 05:58:08,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:08,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/ee2dc6a4004042b291ede6bbbb3f2d07 2023-04-06 05:58:08,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/0cfe32c26ec4491e9dc97765dbf366df 2023-04-06 05:58:08,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/cc36e62be5b04843be380a92711adef8 2023-04-06 05:58:08,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/345c0356af9647b29420fb29c83fcfb6 2023-04-06 05:58:08,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/eca15d37bb6244feaadb501e105da83c 2023-04-06 05:58:08,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/e94275f383944b13b2b1a1de4521d7e3 2023-04-06 05:58:08,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/1bebadd3e0884e16a1bb3cea766ecfb0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/1bebadd3e0884e16a1bb3cea766ecfb0 2023-04-06 05:58:08,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a94b09af4c95424fb6f78416a4868bb1 2023-04-06 05:58:08,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/aa7feb4a394f4eeb86dd7b2f927dc25f 2023-04-06 05:58:08,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d46344aa572944f592dc95ec633f5892 2023-04-06 05:58:08,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/741f10253ad5472a9f32dbab3ecb3f72 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/741f10253ad5472a9f32dbab3ecb3f72 2023-04-06 05:58:08,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/181c63fe4c13494abe8cb7549c189169 2023-04-06 05:58:08,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/589f397f428c450c9afeb0e7a3275b49 2023-04-06 05:58:08,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7f7a92dc4bcc4702b238db86c34d1e59 2023-04-06 05:58:08,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/7637ca5d7f6c4a218f4eb3525c3f195e 2023-04-06 05:58:08,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/d19300e80bb74cfdb92c9de2b79b47ae 2023-04-06 05:58:08,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a9b98b8eb090420bba653052c707c038 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/a9b98b8eb090420bba653052c707c038 2023-04-06 05:58:08,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bf946866fce5406b976081e11ac6dc85 2023-04-06 05:58:08,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/698f6ca715aa4b078cd512a060c8ff71 2023-04-06 05:58:08,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/f951355890584ccfb3c0eb4b0fd75ee0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/f951355890584ccfb3c0eb4b0fd75ee0 2023-04-06 05:58:08,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/73700cd0d0154060ab6eb43beb8d5287 2023-04-06 05:58:08,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/09e749f14c724a6e8633dbe4253e0ac9 2023-04-06 05:58:08,805 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bb492c4ff8e043689430373983779279 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/bb492c4ff8e043689430373983779279 2023-04-06 05:58:08,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/c2d41981d8a944b6b119388d41490961 2023-04-06 05:58:08,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/6a3d02249ff547ecb50f175d2351d04f 2023-04-06 05:58:08,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/46e66285da7147f5959063a79d01a289, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7dd4d003da274b52b7d39cdf93fecf2b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/081378ab002d4d8aa2fb70ed83f32b93, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a187d25bc3954d5d9727c1ed4e2346ef, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/444901c869c343f68f38e3f4df0675fd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e] to archive 2023-04-06 05:58:08,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:08,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/29f071806c7d435ea3f59589982f5626 2023-04-06 05:58:08,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a299c4abbec340b9a7f3e2509e968858 2023-04-06 05:58:08,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/ef4449850e164c6c94cbc175b47182c6 2023-04-06 05:58:08,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/54168f752a5446289cb997e557f35d75 2023-04-06 05:58:08,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/75f1b004f67d441c90eac85b7337338e 2023-04-06 05:58:08,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/46e66285da7147f5959063a79d01a289 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/46e66285da7147f5959063a79d01a289 2023-04-06 05:58:08,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a203dd7f2fa343af912bada72c3316bd 2023-04-06 05:58:08,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/12ecf193d5a94bb49a363a4819f84871 2023-04-06 05:58:08,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/88570f5c9b0b4766a82632d15c4ed1b9 2023-04-06 05:58:08,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7dd4d003da274b52b7d39cdf93fecf2b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7dd4d003da274b52b7d39cdf93fecf2b 2023-04-06 05:58:08,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/3b49566cf4ca4417b8b44b9082ad0775 2023-04-06 05:58:08,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/c637ed1bc5434e1680df913456f1340e 2023-04-06 05:58:08,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/2d85bfd146c9479ca086000b163e85c4 2023-04-06 05:58:08,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/77a7242dd0cc47e8a7d337a441a48cbb 2023-04-06 05:58:08,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/beb841cbf4ff425bb3fe4e3c8b18e428 2023-04-06 05:58:08,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/081378ab002d4d8aa2fb70ed83f32b93 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/081378ab002d4d8aa2fb70ed83f32b93 2023-04-06 05:58:08,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f7ac1c670cdf4956975ca36df76431a1 2023-04-06 05:58:08,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/b421664c555b4d5b89b1b447dd3b24c0 2023-04-06 05:58:08,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a187d25bc3954d5d9727c1ed4e2346ef to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/a187d25bc3954d5d9727c1ed4e2346ef 2023-04-06 05:58:08,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/7e3a4e7c15af48b3b47dcdf869d56b73 2023-04-06 05:58:08,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/630270e3001a4d88aea35f45a06f6c46 2023-04-06 05:58:08,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/444901c869c343f68f38e3f4df0675fd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/444901c869c343f68f38e3f4df0675fd 2023-04-06 05:58:08,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/f43bd6716ee54f9185f17b4b0b03b38b 2023-04-06 05:58:08,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/49766e8bacb1452fa4922b0648687b63 2023-04-06 05:58:08,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/5e638afc54d5480b9fa874525cc2d28e 2023-04-06 05:58:08,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/5ac30c86236b4c46948052067ad8a070, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/64f8689f60ab4269b0b16c82db58daf7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f0741c71cfc140b8862e846415214395, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/6700088764c64128bf6912ad318822b3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/af5b04fdd0204c6699f12b28a7b0d36b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f] to archive 2023-04-06 05:58:08,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:08,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f66f85d7e1ea409283fbe7b7a48ecbb0 2023-04-06 05:58:08,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9864b4e51efa4384810c941bf7862261 2023-04-06 05:58:08,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/4a4f4efc79824f84a8316248dcf60b5f 2023-04-06 05:58:08,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/343f86545458474290c64d7aa20c42a3 2023-04-06 05:58:08,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/49032be064574e8f9671b700cab6290b 2023-04-06 05:58:08,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/5ac30c86236b4c46948052067ad8a070 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/5ac30c86236b4c46948052067ad8a070 2023-04-06 05:58:08,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/3a6376c83a2049238f2295189c80f9c3 2023-04-06 05:58:08,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/619320cc87cd493eb2e21c266db63b8a 2023-04-06 05:58:08,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ebac708faa464a5fbb8e68f8056db887 2023-04-06 05:58:08,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/64f8689f60ab4269b0b16c82db58daf7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/64f8689f60ab4269b0b16c82db58daf7 2023-04-06 05:58:08,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/aa66afc734f6476b98e5f09ac7576068 2023-04-06 05:58:08,906 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/661227a4bebd405fae339694efe2125b 2023-04-06 05:58:08,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/9078183c2d854dd2bc6d1fd129f6debd 2023-04-06 05:58:08,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/103b8f87990e4af69854c576a531ac7a 2023-04-06 05:58:08,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ddefc88cac2449b887d71b7708bb08ce 2023-04-06 05:58:08,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f0741c71cfc140b8862e846415214395 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/f0741c71cfc140b8862e846415214395 2023-04-06 05:58:08,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/a40dc9292a534236a366b26845ee7736 2023-04-06 05:58:08,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/75f82f89bd144c5b935d0052af50b32f 2023-04-06 05:58:08,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/6700088764c64128bf6912ad318822b3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/6700088764c64128bf6912ad318822b3 2023-04-06 05:58:08,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/84cc8e7a7c784e9dbab954c1d0292875 2023-04-06 05:58:08,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/19974a573a1144df9c1e891b2288350d 2023-04-06 05:58:08,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/af5b04fdd0204c6699f12b28a7b0d36b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/af5b04fdd0204c6699f12b28a7b0d36b 2023-04-06 05:58:08,937 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/1ad6bd742e944c5bb57b9918fe5bc0a7 2023-04-06 05:58:08,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/878dc1bb0a3d4bcfb4c7a8e510c09215 2023-04-06 05:58:08,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/b10338b1bff84444902d74dc78de413f 2023-04-06 05:58:08,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits/844.seqid, newMaxSeqId=844, maxSeqId=4 2023-04-06 05:58:08,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72. 2023-04-06 05:58:08,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for efa7c061e44c974bf8b2c1105ad04a72: 2023-04-06 05:58:08,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:08,971 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=efa7c061e44c974bf8b2c1105ad04a72, regionState=CLOSED 2023-04-06 05:58:08,971 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760688971"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760688971"}]},"ts":"1680760688971"} 2023-04-06 05:58:08,974 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-06 05:58:08,974 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure efa7c061e44c974bf8b2c1105ad04a72, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.6670 sec 2023-04-06 05:58:08,975 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-06 05:58:08,975 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=efa7c061e44c974bf8b2c1105ad04a72, UNASSIGN in 1.6710 sec 2023-04-06 05:58:08,976 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760688975"}]},"ts":"1680760688975"} 2023-04-06 05:58:08,976 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:58:08,979 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:58:08,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8770 sec 2023-04-06 05:58:09,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-06 05:58:09,210 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-06 05:58:09,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:58:09,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,213 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 05:58:09,214 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,219 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:09,221 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits] 2023-04-06 05:58:09,224 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/4d98ec184d6e43e1a64a5a71eca57dca to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/4d98ec184d6e43e1a64a5a71eca57dca 2023-04-06 05:58:09,227 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/596195cd9d684e62afc50d08aa66c59b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/A/596195cd9d684e62afc50d08aa66c59b 2023-04-06 05:58:09,232 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/0c615580b3b84b909bfb6a066d61ff7b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/0c615580b3b84b909bfb6a066d61ff7b 2023-04-06 05:58:09,233 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/bbff6d8407e9433785b9efae36fee5c2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/B/bbff6d8407e9433785b9efae36fee5c2 2023-04-06 05:58:09,238 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/0e0547e21f1049e491013ff9536e6ce3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/0e0547e21f1049e491013ff9536e6ce3 2023-04-06 05:58:09,246 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ec75e2f0e2eb427d8c736fe7f0254ce4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/C/ec75e2f0e2eb427d8c736fe7f0254ce4 2023-04-06 05:58:09,249 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits/844.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72/recovered.edits/844.seqid 2023-04-06 05:58:09,250 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/efa7c061e44c974bf8b2c1105ad04a72 2023-04-06 05:58:09,250 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:58:09,251 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:58:09,252 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 05:58:09,255 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040608aad7e6c4f74a4d8651fe8c95cdda64 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040608aad7e6c4f74a4d8651fe8c95cdda64 2023-04-06 05:58:09,256 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ca56639f4274eb0b6d018c93166bed2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ca56639f4274eb0b6d018c93166bed2 2023-04-06 05:58:09,257 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406158ac4d936f849e2a68d57c740e391cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406158ac4d936f849e2a68d57c740e391cc 2023-04-06 05:58:09,258 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406380e8c819da7420cb95d46b46ea861a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406380e8c819da7420cb95d46b46ea861a1 2023-04-06 05:58:09,259 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063e9ab3b2f77d4e1584418946f985e778 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304063e9ab3b2f77d4e1584418946f985e778 2023-04-06 05:58:09,261 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406455f55b81e65432ba89d8fe1797c325a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406455f55b81e65432ba89d8fe1797c325a 2023-04-06 05:58:09,264 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040647579deb301f42e1abf711cf70c3923e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040647579deb301f42e1abf711cf70c3923e 2023-04-06 05:58:09,265 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e0bd4a61d8d4bd4ad5cdbcb23737ee7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e0bd4a61d8d4bd4ad5cdbcb23737ee7 2023-04-06 05:58:09,266 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406540751482aeb4a69a86ee721d378cad3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406540751482aeb4a69a86ee721d378cad3 2023-04-06 05:58:09,267 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067f1754fa4e014e6fbe2ad37cc845e5e2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304067f1754fa4e014e6fbe2ad37cc845e5e2 2023-04-06 05:58:09,268 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406979e681c0f35455d8d78df2814642b7b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406979e681c0f35455d8d78df2814642b7b 2023-04-06 05:58:09,269 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069ca915a5862e4ae28b08cf88d88652cf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069ca915a5862e4ae28b08cf88d88652cf 2023-04-06 05:58:09,270 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b06122c555014793ad51cec1f38ef9fd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b06122c555014793ad51cec1f38ef9fd 2023-04-06 05:58:09,271 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bc60513ab0994e7f9d0ce043660c2ad4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bc60513ab0994e7f9d0ce043660c2ad4 2023-04-06 05:58:09,272 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bdf8868294ae42c8b06066165978360e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406bdf8868294ae42c8b06066165978360e 2023-04-06 05:58:09,273 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c551fc00dc294858a8d39d0420dbb1e3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c551fc00dc294858a8d39d0420dbb1e3 2023-04-06 05:58:09,275 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c93555a383a74a65bcc2a2fa2cb9ccda to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c93555a383a74a65bcc2a2fa2cb9ccda 2023-04-06 05:58:09,276 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ea7e92944c904eb799924d6d4c1e98cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ea7e92944c904eb799924d6d4c1e98cc 2023-04-06 05:58:09,278 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6b20c43551c4f7db5196423d74f28b5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6b20c43551c4f7db5196423d74f28b5 2023-04-06 05:58:09,280 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6e3804c2df94b56ad94c0f19860e6f9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f6e3804c2df94b56ad94c0f19860e6f9 2023-04-06 05:58:09,286 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff02be66bd6a416894beb9021650964a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ff02be66bd6a416894beb9021650964a 2023-04-06 05:58:09,286 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:58:09,289 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,290 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:58:09,300 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:58:09,301 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,301 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:58:09,301 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760689301"}]},"ts":"9223372036854775807"} 2023-04-06 05:58:09,303 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:58:09,303 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => efa7c061e44c974bf8b2c1105ad04a72, NAME => 'TestAcidGuarantees,,1680760659693.efa7c061e44c974bf8b2c1105ad04a72.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:58:09,303 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:58:09,304 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760689303"}]},"ts":"9223372036854775807"} 2023-04-06 05:58:09,307 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:58:09,309 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,310 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 98 msec 2023-04-06 05:58:09,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-06 05:58:09,315 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-06 05:58:09,333 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=552 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=446 (was 412) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 169), AvailableMemoryMB=4282 (was 3532) - AvailableMemoryMB LEAK? - 2023-04-06 05:58:09,347 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=446, ProcessCount=167, AvailableMemoryMB=4282 2023-04-06 05:58:09,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:58:09,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:58:09,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:09,352 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:58:09,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-06 05:58:09,353 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:58:09,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 05:58:09,359 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:09,360 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a empty. 2023-04-06 05:58:09,360 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:09,360 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:58:09,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 05:58:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 05:58:09,778 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:58:09,781 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 636a24b917856079e5736bb56005308a, NAME => 'TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 636a24b917856079e5736bb56005308a, disabling compactions & flushes 2023-04-06 05:58:09,792 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. after waiting 0 ms 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:09,792 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:09,792 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:09,794 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:58:09,795 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760689795"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760689795"}]},"ts":"1680760689795"} 2023-04-06 05:58:09,796 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:58:09,796 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:58:09,797 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760689797"}]},"ts":"1680760689797"} 2023-04-06 05:58:09,797 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:58:09,800 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, ASSIGN}] 2023-04-06 05:58:09,802 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, ASSIGN 2023-04-06 05:58:09,803 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:58:09,954 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=636a24b917856079e5736bb56005308a, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:09,954 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760689954"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760689954"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760689954"}]},"ts":"1680760689954"} 2023-04-06 05:58:09,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:09,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 05:58:10,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 636a24b917856079e5736bb56005308a, NAME => 'TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:58:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:58:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,111 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,112 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A 2023-04-06 05:58:10,112 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A 2023-04-06 05:58:10,112 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:10,113 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 636a24b917856079e5736bb56005308a columnFamilyName A 2023-04-06 05:58:10,113 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(310): Store=636a24b917856079e5736bb56005308a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:10,113 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,114 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B 2023-04-06 05:58:10,114 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B 2023-04-06 05:58:10,114 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:10,115 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 636a24b917856079e5736bb56005308a columnFamilyName B 2023-04-06 05:58:10,115 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(310): Store=636a24b917856079e5736bb56005308a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:10,115 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,116 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C 2023-04-06 05:58:10,116 DEBUG [StoreOpener-636a24b917856079e5736bb56005308a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C 2023-04-06 05:58:10,116 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:10,117 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 636a24b917856079e5736bb56005308a columnFamilyName C 2023-04-06 05:58:10,117 INFO [StoreOpener-636a24b917856079e5736bb56005308a-1] regionserver.HStore(310): Store=636a24b917856079e5736bb56005308a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:10,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:58:10,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:58:10,122 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 636a24b917856079e5736bb56005308a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67700879, jitterRate=0.00882171094417572}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:58:10,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:10,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., pid=49, masterSystemTime=1680760690107 2023-04-06 05:58:10,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,124 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=636a24b917856079e5736bb56005308a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,124 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760690124"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760690124"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760690124"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760690124"}]},"ts":"1680760690124"} 2023-04-06 05:58:10,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-06 05:58:10,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 in 170 msec 2023-04-06 05:58:10,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-06 05:58:10,127 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, ASSIGN in 326 msec 2023-04-06 05:58:10,127 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:58:10,127 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760690127"}]},"ts":"1680760690127"} 2023-04-06 05:58:10,128 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:58:10,129 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:58:10,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 780 msec 2023-04-06 05:58:10,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-06 05:58:10,459 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-06 05:58:10,460 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x5a7ac4a1 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,464 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846250, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,466 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,468 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,469 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:58:10,472 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:58:10,473 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x682c0b9e to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,478 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@647f95f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,478 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x50d6899d to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,486 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58ae18e2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,487 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x6662e946 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,495 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29c1e76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,495 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x5d7deb76 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,503 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c240146, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,504 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,511 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,512 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x6523d347 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,520 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@209faa3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,520 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x28072326 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,528 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5345748a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,528 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x2d90797d to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,533 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9f61fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,533 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7d3df389 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,545 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@534826c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,545 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x65f69c3f to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:10,553 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77459ab3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:10,563 DEBUG [hconnection-0x1cbc7404-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,563 DEBUG [hconnection-0x1764b762-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,563 DEBUG [hconnection-0x224eb7d9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,564 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,565 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:10,565 DEBUG [hconnection-0x7d8dada6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,566 DEBUG [hconnection-0x48c3ef09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,565 DEBUG [hconnection-0x8c2d9a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,567 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,568 DEBUG [hconnection-0x16d2ee45-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,567 DEBUG [hconnection-0x210234e3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,569 DEBUG [hconnection-0x48dbf21e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,567 DEBUG [hconnection-0x5c510a8a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:10,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:10,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:10,569 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,569 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:10,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:10,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:10,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,570 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,571 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:10,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:10,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:10,572 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:10,572 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,572 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:10,572 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:10,572 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:10,572 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,572 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:10,576 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:10,577 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:10,577 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:10,577 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,577 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,579 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,579 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43210, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,584 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,584 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,586 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,586 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:10,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:10,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c84890f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:10,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:10,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:10,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:58:10,595 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:10,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,595 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:10,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:10,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:10,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:10,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,596 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:10,596 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:10,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:10,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:10,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:10,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:10,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:10,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:10,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:10,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:10,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:10,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:10,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760750602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760750603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760690601, max timestamp=1680760690602Num uniques 859; 2023-04-06 05:58:10,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:10,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760690600, max timestamp=1680760690602Num uniques 859; 2023-04-06 05:58:10,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:58:10,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760750602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760690601, max timestamp=1680760690602Num uniques 859; 2023-04-06 05:58:10,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:10,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:10,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:10,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,605 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:10,605 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:10,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760750611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760750613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,705 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:10,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:10,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:10,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760750704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760750704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760750704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:10,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:10,707 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:10,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:10,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:10,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:10,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:10,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:10,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:10,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:10,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:10,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:10,711 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,711 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,712 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:10,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39534c3f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:10,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:10,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,714 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:10,714 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:10,714 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:10,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:10,715 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:10,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,715 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:10,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:10,717 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:10,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:10,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:10,717 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:10,717 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:10,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,718 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:10,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,721 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:10,721 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:10,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:10,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760750715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760750719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,821 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:10,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:10,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:10,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:10,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:10,824 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:10,824 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,824 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:10,824 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:10,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,825 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:10,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:10,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:10,830 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:10,830 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,830 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:10,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:10,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:10,832 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,832 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:10,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:10,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,841 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,841 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:10,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e7a50d6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:10,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:10,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:10,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:10,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:10,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:10,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,845 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:10,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:10,847 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:10,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:10,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:10,847 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:10,847 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:10,848 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,848 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,848 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:10,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:10,851 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,851 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,851 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:10,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,851 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760750908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760750909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760750909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760750925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:10,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760750925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,961 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:10,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:10,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:10,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:10,963 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:10,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:10,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:10,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,964 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:10,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:10,965 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:10,965 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:10,965 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:10,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:10,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:10,966 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:10,966 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:10,967 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,967 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:10,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:10,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c12777[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:10,968 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:10,969 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,969 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:10,969 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,970 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,970 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:10,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:10,972 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:10,972 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:10,972 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:10,972 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:10,972 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:10,973 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:10,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,973 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:10,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:10,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,974 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:10,975 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:10,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:10,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:10,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:10,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,976 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:10,976 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:10,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:10,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/e767bb3de902486e8d51699da9b6c492 2023-04-06 05:58:11,076 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,079 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,079 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,081 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,081 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,081 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,081 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,082 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,083 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc51c11[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,084 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,085 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,085 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,085 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,088 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,088 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,103 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:11,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,103 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,103 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,203 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,206 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f548502[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,211 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,212 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,214 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,214 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760751211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760751213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760751213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:11,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,220 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,220 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760751227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760751228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,320 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,323 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,324 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,325 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,326 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@334492de[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,333 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,337 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,338 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,338 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:11,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/ec50bc75ec5d499c8b21351f9752c8e9 2023-04-06 05:58:11,438 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,441 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,441 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,441 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,441 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,445 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,445 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,445 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,445 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,446 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e83e15e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,449 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,454 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,455 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,460 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,461 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,466 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,466 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,466 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,466 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,566 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,569 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,570 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,571 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,571 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,571 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,573 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,573 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,573 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,573 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,574 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,574 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a72410b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,580 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,580 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,580 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,580 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,580 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,581 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,583 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,584 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,585 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,585 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,685 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,688 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,688 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,689 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,689 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,689 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,689 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,690 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,690 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,690 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,690 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,690 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,692 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,692 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21f5a3d2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,695 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,695 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,695 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,696 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,699 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,699 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,701 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,703 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,703 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:11,703 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760751720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760751720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760751720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760751733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:11,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760751734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,803 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,806 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:11,808 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:11,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:11,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b00750[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:11,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:11,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:11,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:11,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:11,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:11,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:11,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:11,814 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:11,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:11,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:11,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:11,815 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:11,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:11,815 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:11,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:11,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:11,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:11,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:11,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:11,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:11,819 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:11,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:11,819 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:11,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c0738d38353645c582dcfba56aedf60a 2023-04-06 05:58:11,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/e767bb3de902486e8d51699da9b6c492 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492 2023-04-06 05:58:11,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492, entries=150, sequenceid=15, filesize=11.6 K 2023-04-06 05:58:11,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/ec50bc75ec5d499c8b21351f9752c8e9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9 2023-04-06 05:58:11,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9, entries=150, sequenceid=15, filesize=11.6 K 2023-04-06 05:58:11,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c0738d38353645c582dcfba56aedf60a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a 2023-04-06 05:58:11,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a, entries=150, sequenceid=15, filesize=11.6 K 2023-04-06 05:58:11,870 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=111.82 KB/114500 for 636a24b917856079e5736bb56005308a in 1275ms, sequenceid=15, compaction requested=false 2023-04-06 05:58:11,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:11,919 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:11,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:11,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:11,921 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:11,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:11,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:11,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:11,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:11,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:11,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:11,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:11,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:11,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:11,923 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:11,924 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=111.82 KB heapSize=315.45 KB 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:11,924 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:11,932 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/0bca782557604741b1720600543f5552 2023-04-06 05:58:11,939 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/936b5835e6604d34b3ee873702297dca 2023-04-06 05:58:12,347 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/2499f8678d564ea488083cd2080e64b2 2023-04-06 05:58:12,351 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/0bca782557604741b1720600543f5552 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552 2023-04-06 05:58:12,353 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 05:58:12,354 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/936b5835e6604d34b3ee873702297dca as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca 2023-04-06 05:58:12,356 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 05:58:12,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/2499f8678d564ea488083cd2080e64b2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2 2023-04-06 05:58:12,360 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2, entries=150, sequenceid=48, filesize=11.6 K 2023-04-06 05:58:12,360 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~111.82 KB/114500, heapSize ~315.41 KB/322976, currentSize=0 B/0 for 636a24b917856079e5736bb56005308a in 436ms, sequenceid=48, compaction requested=false 2023-04-06 05:58:12,360 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:12,360 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,360 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:12,360 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:12,360 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,361 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:12,361 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:12,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:12,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a899c9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:12,428 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:12,428 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,428 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,428 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:12,428 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:12,428 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,429 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:12,429 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:12,429 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,429 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:12,429 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,429 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:12,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:12,431 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:12,431 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:12,431 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:12,431 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:12,431 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:12,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,436 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,436 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:12,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:12,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,437 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:12,437 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:12,537 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:12,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:12,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:12,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:12,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:12,539 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:12,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:12,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:12,540 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,540 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,541 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,541 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,541 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,541 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:12,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:12,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:12,542 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:12,542 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:12,543 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,543 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:12,543 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:12,543 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:12,543 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:12,544 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:12,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1aee2e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:12,545 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:12,545 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,546 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,546 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,546 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:12,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:12,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:12,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:12,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,547 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:12,547 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,547 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:12,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:12,549 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:12,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:12,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:12,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:12,549 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:12,550 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,550 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,550 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,550 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,551 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,553 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:12,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,553 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:12,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,553 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:12,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:12,653 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:12,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:12,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:12,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:12,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:12,656 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:12,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:12,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:12,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,657 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,657 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,657 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,657 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:12,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:12,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:12,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:12,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:12,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:12,659 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,659 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:12,659 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:12,659 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:12,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:12,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:12,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:12,661 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:12,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63fd3b2c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:12,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:12,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:12,665 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,665 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:12,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:12,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:12,667 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:12,667 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:12,667 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:12,668 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:12,668 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:12,669 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,669 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,669 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,671 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:12,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:12,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,673 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,673 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:12,673 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:12,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:12,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:12,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:12,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760752752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760692749, max timestamp=1680760692757Num uniques 859; 2023-04-06 05:58:12,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:58:12,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760692749, max timestamp=1680760692757Num uniques 859; 2023-04-06 05:58:12,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:58:12,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680760692749, max timestamp=1680760692757Num uniques 859; 2023-04-06 05:58:12,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-06 05:58:12,773 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:12,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:12,777 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:12,777 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,778 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:12,778 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:12,778 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,778 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:12,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:12,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:12,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:12,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:12,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:12,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:12,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:12,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:12,780 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,781 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:12,781 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:12,781 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:12,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:12,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:12,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:12,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:12,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c13a84[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:12,783 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:12,783 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,784 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,784 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:12,784 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:12,784 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:12,784 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:12,785 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:12,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:12,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,788 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:12,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:12,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:12,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:12,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:12,788 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:12,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760752783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,789 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,789 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,789 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,791 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:12,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:12,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:12,793 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:12,793 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:12,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760752787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760752788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760752788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760752859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,893 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:12,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:12,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760752889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:12,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:12,895 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:12,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:12,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:12,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:12,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:12,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:12,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:12,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:12,898 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:12,898 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:12,898 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:12,898 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:12,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,899 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:12,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:12,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36449724[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:12,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:12,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:12,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:12,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760752896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:12,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760752897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:12,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760752897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,902 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:12,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:12,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:12,904 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:12,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:12,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:12,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:12,904 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:12,905 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,905 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:12,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,905 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,907 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:12,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:12,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:12,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:12,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:12,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:12,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:12,909 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:12,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:12,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,909 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:12,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:12,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:12,909 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:12,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:13,009 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:13,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:13,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:13,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:13,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:13,012 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:13,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:13,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:13,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,013 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:13,013 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,013 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:13,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:13,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,014 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:13,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:13,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:13,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:13,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:13,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:13,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:13,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e97645c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:13,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:13,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:13,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:13,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:13,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:13,022 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:13,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:13,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:13,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:13,022 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:13,023 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,023 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:13,023 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,024 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:13,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:13,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:13,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:13,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:13,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:13,033 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,033 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:13,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760753065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760753096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760753103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760753103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760753104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,133 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:13,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:13,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:13,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:13,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:13,136 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:13,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:13,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:13,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,137 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:13,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:13,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/16fd1e1e130e48ee8fe01678babc5dbd 2023-04-06 05:58:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:13,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:13,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:13,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:13,139 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:13,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:13,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:13,139 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:13,142 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:13,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4390af9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:13,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:13,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,150 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:13,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:13,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:13,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:13,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/1bc9c247e66e4ec6b9b88c67071387b4 2023-04-06 05:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:13,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:13,158 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:13,161 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:13,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:13,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:13,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:13,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:13,161 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:13,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c69de0dc3b374de6ac6c1c5fa68f0487 2023-04-06 05:58:13,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:13,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:13,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:13,168 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:13,169 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/16fd1e1e130e48ee8fe01678babc5dbd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd 2023-04-06 05:58:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:13,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd, entries=200, sequenceid=61, filesize=13.9 K 2023-04-06 05:58:13,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/1bc9c247e66e4ec6b9b88c67071387b4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4 2023-04-06 05:58:13,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:13,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:13,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:13,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:13,178 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,178 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:13,178 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:13,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c69de0dc3b374de6ac6c1c5fa68f0487 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487 2023-04-06 05:58:13,179 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487, entries=150, sequenceid=61, filesize=11.6 K 2023-04-06 05:58:13,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 636a24b917856079e5736bb56005308a in 453ms, sequenceid=61, compaction requested=true 2023-04-06 05:58:13,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:13,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:13,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:13,184 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:13,184 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:13,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:13,185 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,185 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,185 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=34.8 K 2023-04-06 05:58:13,185 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=37.1 K 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ec50bc75ec5d499c8b21351f9752c8e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760690593 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting e767bb3de902486e8d51699da9b6c492, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760690593 2023-04-06 05:58:13,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 936b5835e6604d34b3ee873702297dca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680760690605 2023-04-06 05:58:13,186 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0bca782557604741b1720600543f5552, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680760690605 2023-04-06 05:58:13,186 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1bc9c247e66e4ec6b9b88c67071387b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692729 2023-04-06 05:58:13,186 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 16fd1e1e130e48ee8fe01678babc5dbd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692728 2023-04-06 05:58:13,191 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:13,192 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#349 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:13,224 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/818de9bce03c40a3ade8ab7abe3cd059 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/818de9bce03c40a3ade8ab7abe3cd059 2023-04-06 05:58:13,227 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into 818de9bce03c40a3ade8ab7abe3cd059(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:13,227 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:13,227 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760693184; duration=0sec 2023-04-06 05:58:13,227 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:13,227 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:13,229 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:13,229 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:13,229 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,229 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=34.8 K 2023-04-06 05:58:13,229 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c0738d38353645c582dcfba56aedf60a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680760690593 2023-04-06 05:58:13,230 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/4e3443211ddd4f3a905ed2ac7090eb04 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4e3443211ddd4f3a905ed2ac7090eb04 2023-04-06 05:58:13,230 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2499f8678d564ea488083cd2080e64b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680760690605 2023-04-06 05:58:13,230 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c69de0dc3b374de6ac6c1c5fa68f0487, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692729 2023-04-06 05:58:13,233 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into 4e3443211ddd4f3a905ed2ac7090eb04(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:13,233 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:13,233 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760693184; duration=0sec 2023-04-06 05:58:13,233 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:13,237 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#350 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:13,251 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c9309ca0f34d49ca9b7db116d3a5398e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c9309ca0f34d49ca9b7db116d3a5398e 2023-04-06 05:58:13,256 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into c9309ca0f34d49ca9b7db116d3a5398e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:13,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:13,256 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760693184; duration=0sec 2023-04-06 05:58:13,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:13,278 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:13,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:13,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:13,281 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:13,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:13,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:13,282 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,282 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,283 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,283 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:13,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:13,283 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,283 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:13,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:13,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:13,285 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:13,285 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:13,285 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:13,285 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:13,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:13,373 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:13,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760753395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760753401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760753408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760753409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760753409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760753502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,696 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/d36ae741b29d43948d51e7c936513be2 2023-04-06 05:58:13,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760753705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760753910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760753918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760753920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:13,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:13,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760753920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760754011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,104 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/d2474b502f164dc2af540f3d4b952e6e 2023-04-06 05:58:14,491 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:58:14,512 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/4afbd0a441b14815a6d188a23b512b78 2023-04-06 05:58:14,516 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/d36ae741b29d43948d51e7c936513be2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2 2023-04-06 05:58:14,519 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 05:58:14,520 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/d2474b502f164dc2af540f3d4b952e6e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e 2023-04-06 05:58:14,523 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 05:58:14,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/4afbd0a441b14815a6d188a23b512b78 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78 2023-04-06 05:58:14,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760754519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,526 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78, entries=150, sequenceid=99, filesize=11.6 K 2023-04-06 05:58:14,527 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 636a24b917856079e5736bb56005308a in 1242ms, sequenceid=99, compaction requested=false 2023-04-06 05:58:14,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:14,527 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:14,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:14,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:14,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:14,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:14,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:14,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:14,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f005e0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:14,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:14,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:14,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:14,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:14,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:14,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:14,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:14,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:14,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:14,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:14,533 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:14,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:14,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:14,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:14,533 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:14,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:14,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:14,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:14,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:14,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:14,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:14,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:14,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:14,537 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:14,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:14,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,537 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:14,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:14,637 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:14,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:14,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:14,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:14,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:14,639 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:14,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:14,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:14,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:14,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:14,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:14,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:14,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:14,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:14,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:14,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:14,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:14,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:14,642 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:14,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:14,648 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/4b9fdab1983440ef86a6b100e6a0fdfb 2023-04-06 05:58:14,655 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/3b4ff123d82042208aad704452b45ae9 2023-04-06 05:58:14,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:14,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:14,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760694943, max timestamp=1680760694954Num uniques 859; 2023-04-06 05:58:14,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:58:14,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760694943, max timestamp=1680760694954Num uniques 859; 2023-04-06 05:58:14,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:58:14,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680760694943, max timestamp=1680760694954Num uniques 859; 2023-04-06 05:58:14,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-06 05:58:14,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760754974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760754974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760754978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:14,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:14,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760754979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,062 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/a0eea39427bd4c35aeca4ce201fe8363 2023-04-06 05:58:15,065 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/4b9fdab1983440ef86a6b100e6a0fdfb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb 2023-04-06 05:58:15,068 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb, entries=150, sequenceid=109, filesize=11.6 K 2023-04-06 05:58:15,068 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/3b4ff123d82042208aad704452b45ae9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9 2023-04-06 05:58:15,071 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9, entries=150, sequenceid=109, filesize=11.6 K 2023-04-06 05:58:15,071 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/a0eea39427bd4c35aeca4ce201fe8363 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363 2023-04-06 05:58:15,074 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363, entries=150, sequenceid=109, filesize=11.6 K 2023-04-06 05:58:15,074 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 636a24b917856079e5736bb56005308a in 432ms, sequenceid=109, compaction requested=true 2023-04-06 05:58:15,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:15,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,076 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,076 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1f546d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,078 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,080 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,080 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,081 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,082 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,083 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,083 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,083 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:15,084 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,084 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:15,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-06 05:58:15,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:15,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:15,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:15,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:15,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:15,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760755093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760755095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760755096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760755097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,184 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,194 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,195 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760755204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760755204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760755204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760755204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,215 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,217 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,217 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,219 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,219 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9f7cf6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,221 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,223 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,223 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,223 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,223 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,223 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,225 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,226 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:15,226 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,226 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,326 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,331 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,331 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,331 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,331 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,331 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333dcf3e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,335 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,337 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,337 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,340 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,342 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,342 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760755414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760755414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760755415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760755415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,442 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,446 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,447 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,449 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bba13bc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,451 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,452 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,454 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,454 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,455 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,455 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,455 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:15,458 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,458 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,458 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,458 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/2dd5b439e63647aa9a2d5ca5f27eef87 2023-04-06 05:58:15,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/3b4943043ddf49cb996fd980d20f30c1 2023-04-06 05:58:15,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760755534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,558 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,561 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,562 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,564 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b369e9a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,568 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,570 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,570 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,574 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,574 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,575 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,575 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,675 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,678 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,679 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,680 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,680 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,680 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,680 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f4a21c6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,682 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,682 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,683 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,683 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,686 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,686 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,687 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,689 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,691 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,692 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,692 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760755720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760755721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760755721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:15,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760755723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,792 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,795 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,796 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,796 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:15,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,797 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,797 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,797 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,797 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3512e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,800 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,800 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,800 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,803 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,803 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,804 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,804 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,804 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,808 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,808 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,809 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/0972670a933c4d44a1a923543b1ad3d3 2023-04-06 05:58:15,909 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:15,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:15,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:15,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:15,912 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:15,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:15,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:15,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/2dd5b439e63647aa9a2d5ca5f27eef87 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87 2023-04-06 05:58:15,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:15,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:15,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:15,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:15,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87, entries=200, sequenceid=145, filesize=14.1 K 2023-04-06 05:58:15,916 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,916 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:15,916 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:15,916 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,916 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:15,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:15,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:15,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:15,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/3b4943043ddf49cb996fd980d20f30c1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1 2023-04-06 05:58:15,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 05:58:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:15,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394f953b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:15,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:15,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/0972670a933c4d44a1a923543b1ad3d3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3 2023-04-06 05:58:15,921 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,921 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:15,921 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:15,921 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:15,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:15,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,922 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:15,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:15,923 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:15,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:15,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:15,924 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:15,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3, entries=150, sequenceid=145, filesize=11.8 K 2023-04-06 05:58:15,924 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:15,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 636a24b917856079e5736bb56005308a in 837ms, sequenceid=145, compaction requested=true 2023-04-06 05:58:15,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:15,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:15,924 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:15,924 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:15,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:15,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:15,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:15,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,925 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:15,926 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50187 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:15,926 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:15,926 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:15,926 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,926 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,926 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,926 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/818de9bce03c40a3ade8ab7abe3cd059, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=46.7 K 2023-04-06 05:58:15,926 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4e3443211ddd4f3a905ed2ac7090eb04, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=49.0 K 2023-04-06 05:58:15,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:15,927 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 818de9bce03c40a3ade8ab7abe3cd059, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692729 2023-04-06 05:58:15,927 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4e3443211ddd4f3a905ed2ac7090eb04, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692729 2023-04-06 05:58:15,927 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d2474b502f164dc2af540f3d4b952e6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760692777 2023-04-06 05:58:15,927 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d36ae741b29d43948d51e7c936513be2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760692777 2023-04-06 05:58:15,927 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3b4ff123d82042208aad704452b45ae9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760693385 2023-04-06 05:58:15,928 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3b4943043ddf49cb996fd980d20f30c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:15,928 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4b9fdab1983440ef86a6b100e6a0fdfb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760693385 2023-04-06 05:58:15,928 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2dd5b439e63647aa9a2d5ca5f27eef87, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,939 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#360 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:15,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:15,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:15,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:15,944 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:15,945 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:15,945 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:15,945 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:15,947 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#361 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:15,955 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/8eb03a28e0674606b0f027906190c476 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/8eb03a28e0674606b0f027906190c476 2023-04-06 05:58:15,960 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into 8eb03a28e0674606b0f027906190c476(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:15,960 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:15,960 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=12, startTime=1680760695924; duration=0sec 2023-04-06 05:58:15,960 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:15,960 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:15,961 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/bf15fd8dc1154d67bdf84c72b63dc932 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/bf15fd8dc1154d67bdf84c72b63dc932 2023-04-06 05:58:15,961 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:15,962 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:15,962 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:15,962 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c9309ca0f34d49ca9b7db116d3a5398e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=46.7 K 2023-04-06 05:58:15,962 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c9309ca0f34d49ca9b7db116d3a5398e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680760692729 2023-04-06 05:58:15,962 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4afbd0a441b14815a6d188a23b512b78, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680760692777 2023-04-06 05:58:15,962 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a0eea39427bd4c35aeca4ce201fe8363, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1680760693385 2023-04-06 05:58:15,963 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0972670a933c4d44a1a923543b1ad3d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:15,966 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into bf15fd8dc1154d67bdf84c72b63dc932(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:15,966 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:15,966 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=12, startTime=1680760695924; duration=0sec 2023-04-06 05:58:15,966 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:15,970 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#362 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:16,045 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:16,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:16,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:16,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:16,048 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:16,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:16,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:16,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:16,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:16,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:16,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:16,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:16,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:16,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:16,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:16,051 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:16,051 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:16,060 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/279d56e131b340e9930470f71813a4c3 2023-04-06 05:58:16,095 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/215b5ddabc9a475a8ffe4dc964406cce 2023-04-06 05:58:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:16,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760696269, max timestamp=1680760696279Num uniques 859; 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760696269, max timestamp=1680760696279Num uniques 859; 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760696269, max timestamp=1680760696279Num uniques 859; 2023-04-06 05:58:16,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760756301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760756302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760756304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760696300, max timestamp=1680760696307Num uniques 1018; 2023-04-06 05:58:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760696300, max timestamp=1680760696307Num uniques 1018; 2023-04-06 05:58:16,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760696300, max timestamp=1680760696307Num uniques 1018; 2023-04-06 05:58:16,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:16,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:16,380 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/935010bcb4994e1883584b759592c0d0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/935010bcb4994e1883584b759592c0d0 2023-04-06 05:58:16,384 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into 935010bcb4994e1883584b759592c0d0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:16,384 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:16,384 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=12, startTime=1680760695925; duration=0sec 2023-04-06 05:58:16,384 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760696352, max timestamp=1680760696388Num uniques 1027; 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760696352, max timestamp=1680760696388Num uniques 1027; 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760696352, max timestamp=1680760696388Num uniques 1027; 2023-04-06 05:58:16,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:16,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760756424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760756425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760696413, max timestamp=1680760696425Num uniques 1086; 2023-04-06 05:58:16,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8509, totalHeapSize=19584, min timestamp=1680760696413, max timestamp=1680760696425Num uniques 1036; 2023-04-06 05:58:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1680760696413, max timestamp=1680760696425Num uniques 1036; 2023-04-06 05:58:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:58:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 05:58:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47439, new segment data size=8509, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 05:58:16,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760696449, max timestamp=1680760696465Num uniques 1045; 2023-04-06 05:58:16,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:58:16,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760696449, max timestamp=1680760696465Num uniques 1045; 2023-04-06 05:58:16,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:58:16,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760696449, max timestamp=1680760696465Num uniques 1045; 2023-04-06 05:58:16,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:58:16,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760756510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760756513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760756528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760756528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,532 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/0d50340c3b6b4fc994e51b9e572b10dc 2023-04-06 05:58:16,535 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/279d56e131b340e9930470f71813a4c3 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3 2023-04-06 05:58:16,538 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3, entries=150, sequenceid=156, filesize=11.8 K 2023-04-06 05:58:16,538 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/215b5ddabc9a475a8ffe4dc964406cce as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce 2023-04-06 05:58:16,541 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce, entries=150, sequenceid=156, filesize=11.8 K 2023-04-06 05:58:16,541 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/0d50340c3b6b4fc994e51b9e572b10dc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc 2023-04-06 05:58:16,544 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc, entries=150, sequenceid=156, filesize=11.8 K 2023-04-06 05:58:16,544 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 636a24b917856079e5736bb56005308a in 493ms, sequenceid=156, compaction requested=false 2023-04-06 05:58:16,545 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:16,545 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:16,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:16,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:16,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,546 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,546 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:16,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56621ec6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:16,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:16,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,547 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:16,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,548 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:16,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:16,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:16,550 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:16,550 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:16,550 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:16,550 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:16,550 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:16,551 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,551 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,551 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,553 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:16,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:16,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:16,555 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,555 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,555 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:16,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=147.60 KB heapSize=416.77 KB 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:16,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:16,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760756635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760756636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,655 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:16,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:16,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:16,658 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:16,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:16,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:16,658 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,659 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,659 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,660 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:16,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:16,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:16,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:16,660 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,661 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:16,661 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:16,661 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:16,661 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:16,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582851dd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:16,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:16,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:16,664 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,664 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:16,664 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:16,664 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:16,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:16,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:16,666 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:16,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:16,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:16,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:16,666 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:16,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,668 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,670 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:16,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:16,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:16,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,671 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,671 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,671 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:16,671 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,671 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:16,671 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760756732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760756732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760756741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760756741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,771 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:16,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:16,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:16,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:16,774 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:16,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:16,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:16,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,774 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,775 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:16,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:16,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:16,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:16,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:16,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:16,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:16,777 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,777 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:16,777 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:16,777 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:16,777 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:16,778 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,778 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:16,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406a31d7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:16,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:16,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:16,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:16,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,780 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:16,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:16,782 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:16,782 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:16,782 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:16,782 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:16,782 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:16,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,785 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,785 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:16,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,787 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,787 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,787 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:16,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,787 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:16,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:16,887 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:16,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:16,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:16,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:16,891 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:16,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:16,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:16,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:16,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:16,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:16,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:16,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:16,894 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,894 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:16,894 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:16,894 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:16,895 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c982ac[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:16,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:16,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:16,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:16,897 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,897 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:16,899 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:16,899 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:16,899 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:16,899 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:16,899 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:16,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,900 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:16,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:16,900 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:16,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:16,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:16,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:16,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:16,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:16,903 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:16,903 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:16,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760756946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:16,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:16,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760756946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,003 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,006 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,007 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,008 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,008 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,008 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,008 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,010 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439b28fd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,012 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,012 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,012 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,014 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,014 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,016 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,018 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,020 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,020 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,022 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/5ae328250e7647e3ab3bc727e45d4560 2023-04-06 05:58:17,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760757036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760757037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,120 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,123 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,124 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,125 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,125 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,125 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,125 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,125 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,126 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,127 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59a73fea[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,128 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,128 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,128 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,128 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,129 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,128 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,129 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,129 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,131 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,131 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,135 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,135 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,135 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,136 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,236 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,238 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,238 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,238 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,238 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,239 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,240 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,240 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,241 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,241 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,242 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,243 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,243 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d24980[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,244 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,244 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,244 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,244 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,244 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,244 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,244 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,244 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,245 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,245 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,245 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,245 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,247 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,247 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,249 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,253 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,253 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,253 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760757252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760757253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,353 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,355 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,356 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,357 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,357 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,358 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,358 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,358 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,358 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,358 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,359 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,359 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,359 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c77ed4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,362 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,364 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,364 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,364 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,364 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,364 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,365 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,365 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,365 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,367 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,367 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,368 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,368 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,368 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/88589ae4cd0a4bc2816f1e61c0ea16ee 2023-04-06 05:58:17,468 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,471 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,524 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,526 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,526 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,526 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,526 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,528 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b8172d1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,530 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,532 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,532 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,534 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,535 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,536 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,536 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760757542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43214 deadline: 1680760757544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760757548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,553 DEBUG [Thread-2715] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:17,637 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,639 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,639 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,642 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,643 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b394624[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,647 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,648 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,649 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,652 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,652 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,752 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,756 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,757 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,757 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,757 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:17,757 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:17,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:17,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:17,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217d8e8a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:17,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:17,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:17,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:17,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:17,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:17,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760757760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:17,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:17,763 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:17,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:17,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:17,763 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:17,763 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:17,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43166 deadline: 1680760757760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:17,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:17,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:17,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:17,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:17,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,768 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:17,768 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:17,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:17,768 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/5d9a963d7ae1412ca2fd0f4cfa71f96e 2023-04-06 05:58:17,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/5ae328250e7647e3ab3bc727e45d4560 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560 2023-04-06 05:58:17,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560, entries=200, sequenceid=263, filesize=14.3 K 2023-04-06 05:58:17,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/88589ae4cd0a4bc2816f1e61c0ea16ee as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee 2023-04-06 05:58:17,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 05:58:17,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/5d9a963d7ae1412ca2fd0f4cfa71f96e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e 2023-04-06 05:58:17,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e, entries=150, sequenceid=263, filesize=11.9 K 2023-04-06 05:58:17,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~416.72 KB/426720, currentSize=33.54 KB/34350 for 636a24b917856079e5736bb56005308a in 1236ms, sequenceid=263, compaction requested=true 2023-04-06 05:58:17,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:17,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:17,856 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:17,856 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:17,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:17,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:17,856 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38938 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:17,857 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:17,857 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:17,857 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:17,857 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,857 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,857 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/8eb03a28e0674606b0f027906190c476, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.0 K 2023-04-06 05:58:17,857 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/bf15fd8dc1154d67bdf84c72b63dc932, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=35.6 K 2023-04-06 05:58:17,857 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 8eb03a28e0674606b0f027906190c476, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:17,857 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bf15fd8dc1154d67bdf84c72b63dc932, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:17,858 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 279d56e131b340e9930470f71813a4c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680760695088 2023-04-06 05:58:17,858 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 215b5ddabc9a475a8ffe4dc964406cce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680760695088 2023-04-06 05:58:17,858 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5ae328250e7647e3ab3bc727e45d4560, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:17,858 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 88589ae4cd0a4bc2816f1e61c0ea16ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:17,862 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:17,862 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#370 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:17,868 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:17,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:17,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:17,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:17,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:17,871 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:17,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:17,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:17,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,873 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:17,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:17,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:17,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:17,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:17,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:17,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:17,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:17,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:17,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:17,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:17,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:17,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:17,875 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:17,875 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:17,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:18,281 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f327de7d85e840b5870bfb244f94755c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f327de7d85e840b5870bfb244f94755c 2023-04-06 05:58:18,281 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/686385adf2374b5dadb481cebd2b96ba as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/686385adf2374b5dadb481cebd2b96ba 2023-04-06 05:58:18,284 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f2a6b7797d63436f87f0eb98a063e00f 2023-04-06 05:58:18,285 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into 686385adf2374b5dadb481cebd2b96ba(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:18,285 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into f327de7d85e840b5870bfb244f94755c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:18,285 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:18,285 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760697856; duration=0sec 2023-04-06 05:58:18,285 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:18,285 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760697856; duration=0sec 2023-04-06 05:58:18,286 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:18,286 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:18,285 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:18,286 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:18,286 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:18,287 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:18,287 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/935010bcb4994e1883584b759592c0d0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=35.6 K 2023-04-06 05:58:18,287 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 935010bcb4994e1883584b759592c0d0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680760694963 2023-04-06 05:58:18,287 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0d50340c3b6b4fc994e51b9e572b10dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680760695088 2023-04-06 05:58:18,288 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5d9a963d7ae1412ca2fd0f4cfa71f96e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:18,292 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#373 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:18,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:18,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760698582, max timestamp=1680760698592Num uniques 859; 2023-04-06 05:58:18,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:18,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760698582, max timestamp=1680760698592Num uniques 859; 2023-04-06 05:58:18,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:18,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760698582, max timestamp=1680760698592Num uniques 859; 2023-04-06 05:58:18,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760698632, max timestamp=1680760698644Num uniques 1068; 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760698632, max timestamp=1680760698644Num uniques 1068; 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680760698632, max timestamp=1680760698644Num uniques 1068; 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:58:18,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-06 05:58:18,692 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/4d2b19ab0217405997fae7ac7c378a78 2023-04-06 05:58:18,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760698680, max timestamp=1680760698700Num uniques 1077; 2023-04-06 05:58:18,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760758701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760698680, max timestamp=1680760698700Num uniques 1077; 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680760698680, max timestamp=1680760698700Num uniques 1077; 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:58:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-06 05:58:18,705 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/6c87ee003a2346d7ba56ca9e9cfa9390 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/6c87ee003a2346d7ba56ca9e9cfa9390 2023-04-06 05:58:18,708 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into 6c87ee003a2346d7ba56ca9e9cfa9390(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:18,708 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:18,708 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760697856; duration=0sec 2023-04-06 05:58:18,708 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760698749, max timestamp=1680760698765Num uniques 1086; 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760698749, max timestamp=1680760698765Num uniques 1086; 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760698749, max timestamp=1680760698765Num uniques 1086; 2023-04-06 05:58:18,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-06 05:58:18,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760758798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760698781, max timestamp=1680760698800Num uniques 1045; 2023-04-06 05:58:18,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 05:58:18,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760698781, max timestamp=1680760698800Num uniques 1045; 2023-04-06 05:58:18,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760698781, max timestamp=1680760698800Num uniques 1045; 2023-04-06 05:58:18,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 05:58:18,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 05:58:18,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760758826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760698822, max timestamp=1680760698828Num uniques 1054; 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760698822, max timestamp=1680760698828Num uniques 1054; 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760698822, max timestamp=1680760698828Num uniques 1054; 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 05:58:18,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760698875, max timestamp=1680760698890Num uniques 1063; 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760698875, max timestamp=1680760698890Num uniques 1063; 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760698875, max timestamp=1680760698890Num uniques 1063; 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:18,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:18,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760758930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760758932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760698920, max timestamp=1680760698940Num uniques 1072; 2023-04-06 05:58:18,941 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:18,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760698920, max timestamp=1680760698940Num uniques 1072; 2023-04-06 05:58:18,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:18,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760698920, max timestamp=1680760698940Num uniques 1072; 2023-04-06 05:58:18,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:18,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43202 deadline: 1680760758994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:18,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680760698982, max timestamp=1680760698994Num uniques 1081; 2023-04-06 05:58:18,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680760698982, max timestamp=1680760698994Num uniques 1081; 2023-04-06 05:58:18,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 05:58:18,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 05:58:18,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680760698982, max timestamp=1680760698994Num uniques 1081; 2023-04-06 05:58:18,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680760699039, max timestamp=1680760699071Num uniques 1090; 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680760699039, max timestamp=1680760699071Num uniques 1090; 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680760699039, max timestamp=1680760699071Num uniques 1090; 2023-04-06 05:58:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-06 05:58:19,099 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/2bdff987de784a769b88f7416e23befc 2023-04-06 05:58:19,102 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f2a6b7797d63436f87f0eb98a063e00f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f 2023-04-06 05:58:19,105 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f, entries=150, sequenceid=271, filesize=11.9 K 2023-04-06 05:58:19,105 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/4d2b19ab0217405997fae7ac7c378a78 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78 2023-04-06 05:58:19,108 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78, entries=150, sequenceid=271, filesize=11.9 K 2023-04-06 05:58:19,109 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/2bdff987de784a769b88f7416e23befc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc 2023-04-06 05:58:19,112 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc, entries=150, sequenceid=271, filesize=11.9 K 2023-04-06 05:58:19,112 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=134.18 KB/137400 for 636a24b917856079e5736bb56005308a in 1237ms, sequenceid=271, compaction requested=false 2023-04-06 05:58:19,112 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:19,112 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,112 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,112 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,112 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,114 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf665af[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,115 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,115 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,115 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,115 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,115 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,116 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:19,116 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=140.89 KB heapSize=397.78 KB 2023-04-06 05:58:19,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,117 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,117 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,117 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680760699102, max timestamp=1680760699115Num uniques 1099; 2023-04-06 05:58:19,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680760699102, max timestamp=1680760699115Num uniques 1099; 2023-04-06 05:58:19,117 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-06 05:58:19,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-06 05:58:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:19,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:19,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:19,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:19,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:19,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,120 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,121 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,121 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,121 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,122 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,123 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,125 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,126 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,126 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,126 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,126 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:19,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760759145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760759148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760759149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760759150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,226 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,229 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,229 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,230 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,230 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,231 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,231 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,232 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,232 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,232 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,233 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf19bfa[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,234 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,234 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,234 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,235 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,235 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,235 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,235 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,237 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,237 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,237 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,237 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,237 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,238 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,238 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,238 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,238 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,239 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,242 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,242 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,242 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,242 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760759255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760759255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760759255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760759256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,342 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,345 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,345 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,346 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,346 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,347 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,349 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,349 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,349 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,349 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,349 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,350 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,350 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ac8e4c6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,351 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,352 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,352 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,352 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,353 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,354 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,355 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,355 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,355 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,356 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,356 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,356 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,357 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,359 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,359 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,359 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,359 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,459 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,462 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,462 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,463 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,463 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,464 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,464 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,464 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,465 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,465 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,465 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,465 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,466 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@770c24cb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,467 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,467 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,467 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,468 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,468 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,470 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,470 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,470 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,470 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,470 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,470 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,470 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760759464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,471 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,471 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760759464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760759465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760759465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,472 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,473 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,474 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,474 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/d716686592fb48feada17d8e7a9b3ed0 2023-04-06 05:58:19,574 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,576 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,576 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,579 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,579 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,580 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,580 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fffec98[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,582 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,582 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,582 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,583 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,583 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,585 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,585 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,585 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,585 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,586 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,587 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,588 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,589 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:19,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,589 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,589 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,589 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,690 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,692 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,692 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,692 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,692 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,693 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,695 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,695 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,696 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,696 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d682d6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,698 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,698 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,700 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,700 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,700 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,701 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,701 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,701 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,701 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,705 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,706 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,705 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,706 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43190 deadline: 1680760759772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760759773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760759773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:19,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760759774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,806 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,808 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,810 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,810 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,810 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,810 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,811 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,811 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,812 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,813 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c5b3de[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,819 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,819 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,820 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,821 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,822 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,822 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:19,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,823 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,823 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,923 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:19,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:19,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:19,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:19,926 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:19,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:19,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:19,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,927 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:19,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:19,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:19,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:19,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:19,928 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,928 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:19,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,929 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/066eb74ad1b347c9bc92b7ac8e8fc089 2023-04-06 05:58:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:19,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f9f1d1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:19,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:19,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:19,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:19,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:19,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,934 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:19,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:19,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:19,941 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:19,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:19,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:19,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:19,941 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:19,942 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,942 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,942 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:19,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/680a4d682a9442349783d836b6b1cc8f 2023-04-06 05:58:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,943 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,944 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:19,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/d716686592fb48feada17d8e7a9b3ed0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0 2023-04-06 05:58:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:19,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:19,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:19,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:19,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:19,949 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:19,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,950 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:19,950 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:19,950 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:19,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0, entries=200, sequenceid=466, filesize=14.3 K 2023-04-06 05:58:19,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/066eb74ad1b347c9bc92b7ac8e8fc089 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089 2023-04-06 05:58:19,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089, entries=150, sequenceid=466, filesize=11.9 K 2023-04-06 05:58:19,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/680a4d682a9442349783d836b6b1cc8f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f 2023-04-06 05:58:19,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f, entries=150, sequenceid=466, filesize=11.9 K 2023-04-06 05:58:19,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~191.08 KB/195664, currentSize=114.05 KB/116790 for 636a24b917856079e5736bb56005308a in 848ms, sequenceid=466, compaction requested=true 2023-04-06 05:58:19,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:19,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:19,964 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:19,964 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:19,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:19,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:19,965 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,965 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,965 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f327de7d85e840b5870bfb244f94755c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.4 K 2023-04-06 05:58:19,965 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/686385adf2374b5dadb481cebd2b96ba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.0 K 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f327de7d85e840b5870bfb244f94755c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:19,965 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 686385adf2374b5dadb481cebd2b96ba, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:19,966 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f2a6b7797d63436f87f0eb98a063e00f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680760696622 2023-04-06 05:58:19,966 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4d2b19ab0217405997fae7ac7c378a78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680760696622 2023-04-06 05:58:19,966 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d716686592fb48feada17d8e7a9b3ed0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:19,966 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 066eb74ad1b347c9bc92b7ac8e8fc089, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:19,971 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:19,973 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#379 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:19,982 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/802e730daca64889ba0b67bf8cb1603f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/802e730daca64889ba0b67bf8cb1603f 2023-04-06 05:58:19,984 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/ba0d9cf7146f42d3be70b2179acde3cf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ba0d9cf7146f42d3be70b2179acde3cf 2023-04-06 05:58:19,985 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into 802e730daca64889ba0b67bf8cb1603f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:19,986 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:19,986 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760699964; duration=0sec 2023-04-06 05:58:19,986 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:19,986 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:19,997 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:19,997 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:19,997 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:19,997 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/6c87ee003a2346d7ba56ca9e9cfa9390, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.0 K 2023-04-06 05:58:19,997 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 6c87ee003a2346d7ba56ca9e9cfa9390, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680760696505 2023-04-06 05:58:19,998 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2bdff987de784a769b88f7416e23befc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680760696622 2023-04-06 05:58:19,998 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 680a4d682a9442349783d836b6b1cc8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:19,999 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into ba0d9cf7146f42d3be70b2179acde3cf(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:19,999 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:19,999 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760699964; duration=0sec 2023-04-06 05:58:19,999 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:20,007 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:20,026 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/1fd888c2a8764f56af843bc7b2610ef1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/1fd888c2a8764f56af843bc7b2610ef1 2023-04-06 05:58:20,029 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into 1fd888c2a8764f56af843bc7b2610ef1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:20,029 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:20,029 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760699964; duration=0sec 2023-04-06 05:58:20,029 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:20,050 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:20,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:20,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:20,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:20,055 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:20,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:20,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:20,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:20,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:20,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:20,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:20,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:20,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:20,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:20,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:20,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:20,058 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:20,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:20,059 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:20,059 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:20,059 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:20,060 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:20,060 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,071 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f83a3c360fdd4d0591ae16cdb6f202e9 2023-04-06 05:58:20,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:20,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:20,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760760288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760760288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760760290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760760291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760760392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760760395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760760396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760760400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,484 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/653eed6351d04f76a61343d014bd7175 2023-04-06 05:58:20,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760760596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760760608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760760608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760760608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,893 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/86dc4f0802b149d391ab160a5d51d5f2 2023-04-06 05:58:20,896 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f83a3c360fdd4d0591ae16cdb6f202e9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9 2023-04-06 05:58:20,899 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9, entries=150, sequenceid=489, filesize=11.9 K 2023-04-06 05:58:20,899 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/653eed6351d04f76a61343d014bd7175 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175 2023-04-06 05:58:20,902 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175, entries=150, sequenceid=489, filesize=11.9 K 2023-04-06 05:58:20,903 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/86dc4f0802b149d391ab160a5d51d5f2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2 2023-04-06 05:58:20,906 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2, entries=150, sequenceid=489, filesize=11.9 K 2023-04-06 05:58:20,907 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 636a24b917856079e5736bb56005308a in 847ms, sequenceid=489, compaction requested=false 2023-04-06 05:58:20,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:20,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:20,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,909 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:20,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:20,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c2501a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:20,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:20,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:20,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:20,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:20,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:20,914 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:20,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:20,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:20,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:20,914 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:20,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:20,916 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:20,916 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:20,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:20,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,938 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:20,939 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:20,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:20,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:20,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:20,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:20,942 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:20,942 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:20,939 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:20,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:20,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760760940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760760941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760760942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:20,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760760942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,042 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,044 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,045 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,045 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,046 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,046 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,048 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,048 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760761047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760761047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760761047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760761048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7feea086[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,051 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,051 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,054 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,054 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,054 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,054 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,054 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,059 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,060 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,060 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:21,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,160 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,162 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,165 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,165 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,165 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,165 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,165 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,166 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773da7e6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,168 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,168 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,168 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,168 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,170 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,170 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,170 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,170 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,170 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,177 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,181 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,181 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,181 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:21,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760761249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760761249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760761250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760761251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,282 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,284 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,284 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,285 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,285 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,285 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,285 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,291 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,291 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,291 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,291 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,291 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b36a7a9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,300 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,301 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,303 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,303 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,305 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,305 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,306 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,306 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,306 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,309 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,309 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/464373430f4d4819b022bb79a8409b6f 2023-04-06 05:58:21,409 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,412 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,416 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,416 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,416 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a756455[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,417 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,417 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,418 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,418 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,418 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,419 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,419 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,421 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,421 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,423 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,424 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:21,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,425 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,426 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,426 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,526 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,528 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,531 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@151f0d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,534 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,537 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,539 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,539 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760761556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760761556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760761556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760761556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43184 deadline: 1680760761588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,593 DEBUG [Thread-2715] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:21,639 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,642 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,643 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,645 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,646 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8d9a7e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,649 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,649 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,651 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,651 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,652 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,655 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,656 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,656 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5f597b155b3641cd83e44b5643609995 2023-04-06 05:58:21,756 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,759 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,759 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,761 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,762 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,763 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24387d69[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,766 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,767 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,768 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,768 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,768 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,772 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,772 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,773 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,773 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,773 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,873 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,876 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,877 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,877 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,877 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,878 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,879 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,880 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cef277[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,881 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,881 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,882 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,882 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,882 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,883 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:21,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:21,885 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:21,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:21,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:21,885 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:21,885 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:21,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,886 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,890 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:21,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:21,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:21,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:21,889 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:21,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,890 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:21,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,890 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:21,990 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:21,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:21,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:21,993 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,993 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,994 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:21,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:21,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:21,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:21,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:21,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:21,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:21,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:21,995 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,995 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:21,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:21,996 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,996 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:21,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0b68b4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:21,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:21,998 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,998 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:21,998 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,998 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:21,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:21,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:21,998 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:21,999 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:21,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:21,999 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:22,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:22,001 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:22,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:22,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:22,001 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:22,001 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:22,001 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,001 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:22,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:22,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:22,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:22,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:22,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:22,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,004 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,004 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:22,004 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:22,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:22,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:22,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:22,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760762060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:22,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760762064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:22,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760762064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:22,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760762064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,104 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:22,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:22,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:22,107 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,107 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,108 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,108 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:22,108 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,108 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:22,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:22,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:22,109 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:22,109 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,109 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:22,109 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:22,109 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,110 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:22,110 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,111 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:22,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:22,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b235901[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:22,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:22,112 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,112 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:22,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:22,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:22,113 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:22,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:22,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:22,116 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:22,116 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:22,117 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:22,117 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:22,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,117 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,118 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:22,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:22,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:22,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:22,121 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:22,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:22,120 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:22,121 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:22,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:22,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/692e0249bbb24858aa3677ecc624566f 2023-04-06 05:58:22,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/464373430f4d4819b022bb79a8409b6f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f 2023-04-06 05:58:22,143 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f, entries=200, sequenceid=502, filesize=14.3 K 2023-04-06 05:58:22,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5f597b155b3641cd83e44b5643609995 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995 2023-04-06 05:58:22,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995, entries=150, sequenceid=502, filesize=11.9 K 2023-04-06 05:58:22,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/692e0249bbb24858aa3677ecc624566f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f 2023-04-06 05:58:22,149 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f, entries=150, sequenceid=502, filesize=11.9 K 2023-04-06 05:58:22,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 636a24b917856079e5736bb56005308a in 1234ms, sequenceid=502, compaction requested=true 2023-04-06 05:58:22,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:22,150 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:22,150 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:22,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:22,151 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:22,151 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/802e730daca64889ba0b67bf8cb1603f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.5 K 2023-04-06 05:58:22,151 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,151 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ba0d9cf7146f42d3be70b2179acde3cf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.1 K 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 802e730daca64889ba0b67bf8cb1603f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:22,151 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ba0d9cf7146f42d3be70b2179acde3cf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:22,152 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f83a3c360fdd4d0591ae16cdb6f202e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680760699147 2023-04-06 05:58:22,152 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 653eed6351d04f76a61343d014bd7175, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680760699147 2023-04-06 05:58:22,152 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 464373430f4d4819b022bb79a8409b6f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:22,152 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5f597b155b3641cd83e44b5643609995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:22,157 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:22,159 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:22,221 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:22,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:22,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:22,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:22,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:22,223 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:22,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:22,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:22,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:22,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,223 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,224 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:22,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:22,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:22,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:22,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:22,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:22,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:22,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:22,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:22,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:22,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:22,225 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:22,225 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,225 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:22,226 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:22,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:22,226 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:22,226 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:22,570 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/efb11da8cbcd48f79f7c110225e270fb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/efb11da8cbcd48f79f7c110225e270fb 2023-04-06 05:58:22,574 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into efb11da8cbcd48f79f7c110225e270fb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:22,574 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:22,574 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760702150; duration=0sec 2023-04-06 05:58:22,574 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:22,574 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:22,575 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:22,575 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:22,575 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:22,575 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/1fd888c2a8764f56af843bc7b2610ef1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.1 K 2023-04-06 05:58:22,575 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1fd888c2a8764f56af843bc7b2610ef1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680760699102 2023-04-06 05:58:22,576 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 86dc4f0802b149d391ab160a5d51d5f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680760699147 2023-04-06 05:58:22,576 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 692e0249bbb24858aa3677ecc624566f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:22,576 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/f7534180c8eb46f0900bf381709f4cfb as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f7534180c8eb46f0900bf381709f4cfb 2023-04-06 05:58:22,580 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:22,580 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into f7534180c8eb46f0900bf381709f4cfb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:22,580 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:22,580 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760702150; duration=0sec 2023-04-06 05:58:22,580 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:22,632 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/3f94c0dd01ce4f61b9ca4a9462f48085 2023-04-06 05:58:22,639 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5a3fbdda59c944a3b270e5f68cb9969a 2023-04-06 05:58:22,987 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/8f217d323d14472593463773972bf2d6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/8f217d323d14472593463773972bf2d6 2023-04-06 05:58:22,991 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into 8f217d323d14472593463773972bf2d6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:22,991 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:22,991 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760702150; duration=0sec 2023-04-06 05:58:22,991 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:23,061 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/4eef58263e60416eb6021b3189cfda0e 2023-04-06 05:58:23,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:23,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,080 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/3f94c0dd01ce4f61b9ca4a9462f48085 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085 2023-04-06 05:58:23,084 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085, entries=150, sequenceid=521, filesize=11.9 K 2023-04-06 05:58:23,084 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5a3fbdda59c944a3b270e5f68cb9969a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a 2023-04-06 05:58:23,087 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a, entries=150, sequenceid=521, filesize=11.9 K 2023-04-06 05:58:23,088 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/4eef58263e60416eb6021b3189cfda0e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e 2023-04-06 05:58:23,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760763081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760763084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760763085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760763086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,092 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e, entries=150, sequenceid=521, filesize=11.9 K 2023-04-06 05:58:23,092 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 636a24b917856079e5736bb56005308a in 866ms, sequenceid=521, compaction requested=false 2023-04-06 05:58:23,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:23,093 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,093 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c97b439[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,096 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,097 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,099 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,099 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,099 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,100 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,100 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,101 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,101 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,104 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,104 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,105 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,104 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:23,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:23,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:23,205 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,207 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,208 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,209 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,209 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,210 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,210 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,210 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,210 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,211 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb8a496[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,213 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,213 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,213 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,214 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,214 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,216 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,216 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760763210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760763212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,220 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,220 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,220 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760763214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760763219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,320 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,322 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,323 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760763319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,323 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760763320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,325 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,325 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,325 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,325 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,326 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38802b3d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,328 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,328 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760763324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,329 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760763325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,330 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,331 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,335 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,336 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,336 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,436 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,438 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,438 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,438 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,438 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,439 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,441 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,441 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,441 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,441 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,442 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@792a967d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,447 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,447 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,447 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,469 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,469 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,469 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,469 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760763525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760763525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760763530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760763531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,570 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,572 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,573 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,574 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,574 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,574 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,575 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,575 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,575 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@789dd31d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,577 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,577 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,578 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,581 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,581 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,581 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,591 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,593 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,594 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,594 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,595 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,595 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/775c00f4d8eb4cde9e637767b5fb4e92 2023-04-06 05:58:23,695 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,697 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,697 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,698 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,698 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,700 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,700 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,700 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,702 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,702 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,703 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,703 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,704 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,704 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738a448c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,706 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,706 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,706 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,706 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,707 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,707 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,707 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,709 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,709 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,710 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,710 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,710 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,711 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,712 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,713 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,713 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,714 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,814 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,816 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,817 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,818 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,819 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,819 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,819 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,819 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,820 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,820 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,820 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,821 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6a5003[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,822 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,822 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,822 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,822 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,822 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,822 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,823 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,825 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,825 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,825 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,826 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,826 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,827 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,829 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,829 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,829 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,829 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760763827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760763828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760763838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:23,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760763838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,930 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:23,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:23,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:23,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:23,932 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:23,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:23,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:23,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,933 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:23,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:23,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:23,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:23,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:23,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:23,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:23,934 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:23,934 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,934 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:23,934 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:23,934 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:23,935 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,935 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,936 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:23,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67debe4d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:23,936 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:23,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,937 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:23,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:23,937 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:23,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:23,938 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,938 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:23,938 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,938 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:23,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:23,939 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:23,939 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:23,939 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:23,940 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:23,940 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:23,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,941 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:23,943 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:23,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:23,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:23,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,943 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:23,943 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:23,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:23,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:23,943 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/fcca6f3efbc44428af0104269156db77 2023-04-06 05:58:24,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/7ff6c0d51672401f924d27ae77dec0c5 2023-04-06 05:58:24,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/775c00f4d8eb4cde9e637767b5fb4e92 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92 2023-04-06 05:58:24,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92, entries=200, sequenceid=538, filesize=14.3 K 2023-04-06 05:58:24,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/fcca6f3efbc44428af0104269156db77 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77 2023-04-06 05:58:24,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77, entries=150, sequenceid=538, filesize=11.9 K 2023-04-06 05:58:24,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/7ff6c0d51672401f924d27ae77dec0c5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5 2023-04-06 05:58:24,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5, entries=150, sequenceid=538, filesize=11.9 K 2023-04-06 05:58:24,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 636a24b917856079e5736bb56005308a in 840ms, sequenceid=538, compaction requested=true 2023-04-06 05:58:24,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:24,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:24,033 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:24,033 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:24,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:24,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:24,034 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:24,034 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:24,034 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:24,034 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/efb11da8cbcd48f79f7c110225e270fb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.6 K 2023-04-06 05:58:24,034 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:24,034 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:24,034 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:24,035 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f7534180c8eb46f0900bf381709f4cfb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.2 K 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting efb11da8cbcd48f79f7c110225e270fb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f7534180c8eb46f0900bf381709f4cfb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 3f94c0dd01ce4f61b9ca4a9462f48085, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760700929 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5a3fbdda59c944a3b270e5f68cb9969a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760700929 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 775c00f4d8eb4cde9e637767b5fb4e92, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:24,035 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting fcca6f3efbc44428af0104269156db77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:24,041 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#396 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:24,041 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:24,043 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:24,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:24,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:24,050 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,050 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:24,051 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:24,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:24,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:24,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:24,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:24,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:24,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:24,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:24,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:24,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:24,054 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:24,055 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:24,055 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:24,055 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:24,055 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-06 05:58:24,055 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:24,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:24,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:24,056 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,060 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/798b9c7230dd44a69141e58f4d8fd31c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/798b9c7230dd44a69141e58f4d8fd31c 2023-04-06 05:58:24,062 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/48567295f074414e87cb206a45cf9f18 2023-04-06 05:58:24,063 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into 798b9c7230dd44a69141e58f4d8fd31c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:24,064 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:24,064 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760704033; duration=0sec 2023-04-06 05:58:24,064 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:24,064 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:24,064 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:24,065 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:24,065 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:24,065 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/8f217d323d14472593463773972bf2d6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.2 K 2023-04-06 05:58:24,065 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 8f217d323d14472593463773972bf2d6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760700285 2023-04-06 05:58:24,065 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4eef58263e60416eb6021b3189cfda0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680760700929 2023-04-06 05:58:24,065 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 7ff6c0d51672401f924d27ae77dec0c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:24,070 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#400 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:24,077 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/e51e589472da4052b965d2daf649cd12 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e51e589472da4052b965d2daf649cd12 2023-04-06 05:58:24,081 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into e51e589472da4052b965d2daf649cd12(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:24,081 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:24,081 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760704033; duration=0sec 2023-04-06 05:58:24,081 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:24,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:24,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760764346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760764346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760764347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760764349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760764452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760764452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760764452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,458 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/426d28d899cd4e2dbbaf5c22d58d5e56 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/426d28d899cd4e2dbbaf5c22d58d5e56 2023-04-06 05:58:24,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,461 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into 426d28d899cd4e2dbbaf5c22d58d5e56(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:24,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760764457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,461 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:24,462 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760704033; duration=0sec 2023-04-06 05:58:24,462 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:24,471 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/29316c9497944d3cb913d2a7d77540e6 2023-04-06 05:58:24,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760764657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760764657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760764657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760764662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,879 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/ab9d16e49519459bbe33c9eac72c4bda 2023-04-06 05:58:24,882 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/48567295f074414e87cb206a45cf9f18 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18 2023-04-06 05:58:24,885 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18, entries=150, sequenceid=556, filesize=11.9 K 2023-04-06 05:58:24,885 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/29316c9497944d3cb913d2a7d77540e6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6 2023-04-06 05:58:24,887 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6, entries=150, sequenceid=556, filesize=11.9 K 2023-04-06 05:58:24,888 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/ab9d16e49519459bbe33c9eac72c4bda as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda 2023-04-06 05:58:24,890 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda, entries=150, sequenceid=556, filesize=11.9 K 2023-04-06 05:58:24,891 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 636a24b917856079e5736bb56005308a in 836ms, sequenceid=556, compaction requested=false 2023-04-06 05:58:24,891 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:24,891 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:24,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:24,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:24,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:24,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:24,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:24,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:24,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cd8c90[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:24,893 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:24,893 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,894 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:24,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:24,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:24,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:24,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:24,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:24,896 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:24,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:24,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:24,898 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:24,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:24,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:24,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:24,898 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:24,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:24,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:24,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,921 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:24,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:24,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:24,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:24,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:24,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:24,922 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:24,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:24,922 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:24,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:24,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:24,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:24,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:24,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760764984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760764984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760764985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:24,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:24,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760764985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,022 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,025 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,026 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,026 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,027 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,027 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,027 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,027 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50420801[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,030 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,030 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,031 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,031 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,034 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,034 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,035 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,036 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,037 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,037 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,037 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,038 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,038 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,038 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,038 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760765090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760765090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760765090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760765091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,138 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,141 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,142 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,143 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,143 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b0a042[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,146 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,146 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,149 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,151 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,151 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,151 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,151 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,151 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,161 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,174 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,174 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,183 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,184 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,184 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,284 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,286 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,286 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,286 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,286 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,287 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,288 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,289 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,289 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,290 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0e53cf[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,293 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,293 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,293 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,297 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760765293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,297 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,298 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,298 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760765293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,298 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,298 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760765296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760765296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/87ab400859de4a3c9b1ea614b550fe7b 2023-04-06 05:58:25,398 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,400 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,400 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,400 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,400 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,401 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,402 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,402 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,405 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,405 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,406 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,406 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47baed12[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,408 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,408 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,409 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,411 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,411 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,412 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,412 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,414 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,415 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,415 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,515 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,518 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,518 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,519 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,519 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,520 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,520 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,520 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,520 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,520 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,521 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,521 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f8f7e4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,523 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,523 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,523 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,525 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,526 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,529 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,529 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760765599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760765600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760765602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:25,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760765603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,630 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,632 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,632 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,633 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,633 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,634 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,635 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,635 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,635 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,635 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a87e5e9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,637 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,638 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,638 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,640 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,640 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,644 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,645 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,745 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,747 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,747 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,748 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,748 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,749 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,749 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,750 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,750 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,750 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,751 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62bf575d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,753 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,753 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,755 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,755 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,762 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,762 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/25c02284b476482c9b8e7a3234eb20c5 2023-04-06 05:58:25,862 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,864 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,866 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,867 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,867 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,867 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,867 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,867 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,868 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@727de465[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,870 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,870 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,871 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,873 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,873 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,873 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,876 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,876 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:25,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,876 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,877 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,877 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,877 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,977 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:25,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:25,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:25,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:25,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:25,979 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:25,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:25,979 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:25,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,981 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,981 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:25,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:25,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:25,981 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:25,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:25,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:25,982 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:25,982 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:25,983 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:25,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327a6a19[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:25,984 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:25,984 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,985 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,985 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:25,985 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,985 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:25,985 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:25,985 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:25,985 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:25,985 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,985 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,986 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:25,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:25,990 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:25,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:25,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:25,990 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:25,990 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:25,994 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,994 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,994 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:25,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:25,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,995 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,996 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:25,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:25,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:25,997 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:25,997 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:25,997 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:25,997 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:26,097 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:26,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:26,100 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,101 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:26,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:26,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:26,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:26,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:26,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:26,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:26,102 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,102 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:26,102 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:26,102 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:26,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:26,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:26,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760766101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:26,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:26,104 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,104 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47a77912[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:26,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:26,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,105 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:26,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:26,105 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:26,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,105 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:26,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:26,107 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:26,107 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:26,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:26,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:26,109 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:26,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:26,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:26,109 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:26,109 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:26,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:26,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760766104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,110 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,110 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,110 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:26,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:26,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:26,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760766107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,111 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:26,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760766110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:26,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:26,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:26,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:26,114 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:26,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:26,114 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:26,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:26,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:26,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:26,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:26,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/435ac4d14630408691fec4efeb050c2b 2023-04-06 05:58:26,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/87ab400859de4a3c9b1ea614b550fe7b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b 2023-04-06 05:58:26,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b, entries=200, sequenceid=574, filesize=14.3 K 2023-04-06 05:58:26,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/25c02284b476482c9b8e7a3234eb20c5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5 2023-04-06 05:58:26,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5, entries=150, sequenceid=574, filesize=11.9 K 2023-04-06 05:58:26,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/435ac4d14630408691fec4efeb050c2b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b 2023-04-06 05:58:26,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b, entries=150, sequenceid=574, filesize=11.9 K 2023-04-06 05:58:26,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 636a24b917856079e5736bb56005308a in 1238ms, sequenceid=574, compaction requested=true 2023-04-06 05:58:26,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:26,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:26,202 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:26,202 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:26,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:26,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:26,203 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:26,203 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/798b9c7230dd44a69141e58f4d8fd31c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.7 K 2023-04-06 05:58:26,203 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,203 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/426d28d899cd4e2dbbaf5c22d58d5e56, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.3 K 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 798b9c7230dd44a69141e58f4d8fd31c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 426d28d899cd4e2dbbaf5c22d58d5e56, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:26,203 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 48567295f074414e87cb206a45cf9f18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680760703208 2023-04-06 05:58:26,204 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 29316c9497944d3cb913d2a7d77540e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680760703208 2023-04-06 05:58:26,204 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 87ab400859de4a3c9b1ea614b550fe7b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:26,204 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 25c02284b476482c9b8e7a3234eb20c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:26,208 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:26,208 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:26,214 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:26,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:26,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:26,218 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:26,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:26,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:26,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:26,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:26,219 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:26,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:26,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:26,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:26,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:26,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:26,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:26,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:26,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:26,221 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:26,221 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:26,221 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/cc767558e18948b6bfb385e8e12ffa40 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/cc767558e18948b6bfb385e8e12ffa40 2023-04-06 05:58:26,224 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into cc767558e18948b6bfb385e8e12ffa40(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:26,225 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:26,225 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760706202; duration=0sec 2023-04-06 05:58:26,225 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:26,225 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:26,226 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:26,226 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:26,226 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:26,226 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e51e589472da4052b965d2daf649cd12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=36.3 K 2023-04-06 05:58:26,226 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e51e589472da4052b965d2daf649cd12, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1680760703070 2023-04-06 05:58:26,227 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ab9d16e49519459bbe33c9eac72c4bda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680760703208 2023-04-06 05:58:26,227 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 435ac4d14630408691fec4efeb050c2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:26,235 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:26,243 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c6f31d18563444658b03a96c901b9a85 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c6f31d18563444658b03a96c901b9a85 2023-04-06 05:58:26,246 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into c6f31d18563444658b03a96c901b9a85(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:26,246 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:26,246 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760706202; duration=0sec 2023-04-06 05:58:26,246 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:26,621 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/9a29ff08c7454a9caa706f456c177863 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/9a29ff08c7454a9caa706f456c177863 2023-04-06 05:58:26,624 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into 9a29ff08c7454a9caa706f456c177863(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:26,624 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:26,624 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760706202; duration=0sec 2023-04-06 05:58:26,624 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:26,637 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/b45bfeca38e34078a284aaf6f0c85fe0 2023-04-06 05:58:26,646 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/51f6e79a90dc406abf52cb97695b0a2f 2023-04-06 05:58:27,055 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/df9cb98a675246b19684cb7426484765 2023-04-06 05:58:27,058 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/b45bfeca38e34078a284aaf6f0c85fe0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0 2023-04-06 05:58:27,068 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:58:27,069 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/51f6e79a90dc406abf52cb97695b0a2f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f 2023-04-06 05:58:27,072 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:58:27,073 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/df9cb98a675246b19684cb7426484765 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765 2023-04-06 05:58:27,077 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765, entries=150, sequenceid=591, filesize=11.9 K 2023-04-06 05:58:27,078 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 636a24b917856079e5736bb56005308a in 857ms, sequenceid=591, compaction requested=false 2023-04-06 05:58:27,078 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:27,078 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,079 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,079 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fbadb04[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,080 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,080 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,081 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,081 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,081 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,082 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,082 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,084 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,084 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,085 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,086 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,088 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,088 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,088 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,088 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:27,118 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:27,118 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:27,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760707139, max timestamp=1680760707146Num uniques 859; 2023-04-06 05:58:27,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:27,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760707139, max timestamp=1680760707146Num uniques 859; 2023-04-06 05:58:27,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:27,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760707139, max timestamp=1680760707146Num uniques 859; 2023-04-06 05:58:27,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:27,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760767166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760767167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760767168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760767173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,188 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,191 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,192 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,192 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,193 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,195 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,195 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,195 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f9525a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,196 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,196 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,197 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,198 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,199 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,199 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,200 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,200 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,200 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,200 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,201 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,203 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,203 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,203 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:27,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760767274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760767274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760767275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760767276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,303 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,305 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b2c2be[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,334 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,336 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,336 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,336 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,436 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,439 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,443 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,443 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,443 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f215033[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,445 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,446 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,448 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,448 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,451 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,452 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,453 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,453 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,453 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760767480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760767480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760767480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760767482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/65f90d8681944f9ba713aca99d2f26c4 2023-04-06 05:58:27,553 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,556 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,557 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,560 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564ac228[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,562 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,563 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,565 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,565 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,565 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,566 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,570 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,570 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,570 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,571 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:27,577 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,578 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,671 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,673 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,673 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,677 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,677 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,678 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,684 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,684 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,684 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,684 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,684 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,691 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,691 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e292139[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,694 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,698 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,698 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,699 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,699 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,699 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,700 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,701 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,702 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,702 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,704 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,704 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,705 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,707 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,707 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,707 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,707 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760767783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760767784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760767787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:27,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760767787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,807 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,810 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,810 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,811 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,811 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,812 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,812 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,814 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,814 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbbbf70[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,815 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,816 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,816 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,816 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,817 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,817 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,818 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,818 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,818 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,819 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,819 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,821 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,823 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,823 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,823 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:27,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,823 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,824 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,924 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:27,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:27,926 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:27,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:27,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:27,926 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,927 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,927 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:27,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:27,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:27,929 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,929 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:27,929 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:27,929 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:27,929 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:27,930 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,930 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:27,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4edafc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:27,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:27,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,932 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:27,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,932 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:27,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:27,932 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:27,933 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,933 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:27,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:27,935 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:27,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:27,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:27,935 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:27,935 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:27,936 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,936 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:27,936 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/f60fb885d3e24be6abd5362ad7636114 2023-04-06 05:58:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:27,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,937 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,939 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:27,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:27,940 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:27,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:27,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:27,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:27,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:27,940 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:27,941 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:27,941 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:28,041 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:28,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:28,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:28,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:28,043 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:28,043 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,043 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:28,043 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:28,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,045 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,045 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:28,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:28,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:28,046 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,046 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:28,046 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:28,046 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:28,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:28,047 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:28,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e15049f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:28,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:28,048 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,048 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:28,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,049 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:28,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:28,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:28,051 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:28,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:28,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:28,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:28,051 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:28,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,053 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,053 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:28,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,055 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:28,055 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,055 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,155 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:28,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:28,158 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:28,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:28,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:28,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,158 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,159 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:28,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:28,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:28,160 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,160 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:28,160 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:28,160 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:28,160 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:28,161 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,161 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,161 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,161 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:28,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2922ca8d[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:28,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:28,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:28,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,164 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:28,164 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,164 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:28,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:28,166 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:28,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:28,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:28,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:28,166 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:28,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,167 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,167 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,168 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:28,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,170 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,170 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,170 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:28,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,170 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:28,271 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:28,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:28,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:28,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:28,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:28,273 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:28,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:28,273 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:28,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,274 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,274 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:28,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:28,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:28,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:28,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:28,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:28,276 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:28,276 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,276 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:28,277 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:28,277 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:28,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:28,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:28,277 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:28,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:28,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd4b0e6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:28,279 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:28,279 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,279 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,279 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:28,281 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,281 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:28,282 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,282 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,283 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:28,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:28,287 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:28,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:28,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:28,287 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:28,287 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:28,288 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:28,288 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:28,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,288 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,289 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,290 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:28,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:28,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:28,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:28,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:28,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:28,292 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:28,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:28,292 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:28,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760768291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760768294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760768294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760768296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,348 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/b3e32e7bd972431fbf977da5ba9b8df9 2023-04-06 05:58:28,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/65f90d8681944f9ba713aca99d2f26c4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4 2023-04-06 05:58:28,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4, entries=200, sequenceid=604, filesize=14.3 K 2023-04-06 05:58:28,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/f60fb885d3e24be6abd5362ad7636114 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114 2023-04-06 05:58:28,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114, entries=100, sequenceid=604, filesize=9.5 K 2023-04-06 05:58:28,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/b3e32e7bd972431fbf977da5ba9b8df9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9 2023-04-06 05:58:28,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9, entries=100, sequenceid=604, filesize=9.5 K 2023-04-06 05:58:28,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 636a24b917856079e5736bb56005308a in 1244ms, sequenceid=604, compaction requested=true 2023-04-06 05:58:28,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:28,363 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:28,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:28,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:28,363 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:28,363 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:28,363 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39744 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/A is initiating minor compaction (all files) 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:28,364 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/A in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/B is initiating minor compaction (all files) 2023-04-06 05:58:28,364 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/9a29ff08c7454a9caa706f456c177863, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=38.8 K 2023-04-06 05:58:28,364 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/B in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,364 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/cc767558e18948b6bfb385e8e12ffa40, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=34.0 K 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 9a29ff08c7454a9caa706f456c177863, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting cc767558e18948b6bfb385e8e12ffa40, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting b45bfeca38e34078a284aaf6f0c85fe0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760704983 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 51f6e79a90dc406abf52cb97695b0a2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760704983 2023-04-06 05:58:28,364 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 65f90d8681944f9ba713aca99d2f26c4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760707113 2023-04-06 05:58:28,365 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f60fb885d3e24be6abd5362ad7636114, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760707113 2023-04-06 05:58:28,368 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#B#compaction#414 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:28,370 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#A#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:28,392 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:28,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:28,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:28,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:28,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:28,394 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:28,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:28,394 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:28,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,395 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,395 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:28,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:28,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:28,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:28,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:28,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:28,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:28,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:28,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:28,397 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:28,397 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:28,776 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/8f05b78ef21c46c6ae119779cd25b69e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/8f05b78ef21c46c6ae119779cd25b69e 2023-04-06 05:58:28,777 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/e4e8917804f045b9984a99b86709056d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e4e8917804f045b9984a99b86709056d 2023-04-06 05:58:28,782 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/B of 636a24b917856079e5736bb56005308a into 8f05b78ef21c46c6ae119779cd25b69e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:28,782 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:28,782 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/A of 636a24b917856079e5736bb56005308a into e4e8917804f045b9984a99b86709056d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:28,782 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/B, priority=13, startTime=1680760708363; duration=0sec 2023-04-06 05:58:28,782 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:28,782 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/A, priority=13, startTime=1680760708363; duration=0sec 2023-04-06 05:58:28,782 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:28,782 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:28,782 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:28,783 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:28,783 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 636a24b917856079e5736bb56005308a/C is initiating minor compaction (all files) 2023-04-06 05:58:28,783 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 636a24b917856079e5736bb56005308a/C in TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:28,783 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c6f31d18563444658b03a96c901b9a85, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp, totalSize=34.0 K 2023-04-06 05:58:28,783 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c6f31d18563444658b03a96c901b9a85, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680760704344 2023-04-06 05:58:28,783 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting df9cb98a675246b19684cb7426484765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680760704983 2023-04-06 05:58:28,784 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b3e32e7bd972431fbf977da5ba9b8df9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760707113 2023-04-06 05:58:28,788 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 636a24b917856079e5736bb56005308a#C#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:28,805 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/b97314e907b84e92922ab3908162c392 2023-04-06 05:58:29,194 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/e0e25cc2d1be4d428486514b25845335 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e0e25cc2d1be4d428486514b25845335 2023-04-06 05:58:29,198 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 636a24b917856079e5736bb56005308a/C of 636a24b917856079e5736bb56005308a into e0e25cc2d1be4d428486514b25845335(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:29,199 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:29,199 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a., storeName=636a24b917856079e5736bb56005308a/C, priority=13, startTime=1680760708363; duration=0sec 2023-04-06 05:58:29,199 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:29,213 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8 2023-04-06 05:58:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:29,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:29,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760769307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760769308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760769309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760769309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760769410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760769412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760769414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760769414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760769614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760769616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760769619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760769619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,623 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c29712cf0b5b4be5a77464c86130b187 2023-04-06 05:58:29,626 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/b97314e907b84e92922ab3908162c392 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b97314e907b84e92922ab3908162c392 2023-04-06 05:58:29,628 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b97314e907b84e92922ab3908162c392, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:58:29,628 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8 2023-04-06 05:58:29,631 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:58:29,632 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/c29712cf0b5b4be5a77464c86130b187 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c29712cf0b5b4be5a77464c86130b187 2023-04-06 05:58:29,636 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c29712cf0b5b4be5a77464c86130b187, entries=150, sequenceid=640, filesize=11.9 K 2023-04-06 05:58:29,636 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 636a24b917856079e5736bb56005308a in 1239ms, sequenceid=640, compaction requested=false 2023-04-06 05:58:29,636 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:29,636 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:29,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:29,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:29,636 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:29,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:29,637 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,637 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:29,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:29,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:29,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570500da[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:29,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:29,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,640 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:29,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,640 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:29,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:29,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:29,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:29,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:29,641 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:29,641 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:29,643 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:29,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:29,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:29,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:29,643 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:29,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,644 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:29,645 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,646 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:29,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:29,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,647 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:29,647 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:29,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:29,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:29,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:29,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,747 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:29,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:29,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:29,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:29,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:29,750 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:29,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:29,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:29,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:29,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,750 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:29,751 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:29,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:29,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:29,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:29,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:29,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:29,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:29,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:29,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:29,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:29,752 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. started... 2023-04-06 05:58:29,752 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:29,752 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:29,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. as already flushing 2023-04-06 05:58:29,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760709938, max timestamp=1680760709944Num uniques 859; 2023-04-06 05:58:29,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:29,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760709938, max timestamp=1680760709944Num uniques 859; 2023-04-06 05:58:29,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760709938, max timestamp=1680760709944Num uniques 859; 2023-04-06 05:58:29,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:29,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:29,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760769970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760769973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760769973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:29,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760769974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760770077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760770079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760770079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760770079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,164 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f45d5f46286b4403b7461bd35a25e297 2023-04-06 05:58:30,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760770282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760770283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760770283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760770284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,562 DEBUG [Thread-2728] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d3df389 to 127.0.0.1:55165 2023-04-06 05:58:30,562 DEBUG [Thread-2728] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:30,564 DEBUG [Thread-2724] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28072326 to 127.0.0.1:55165 2023-04-06 05:58:30,564 DEBUG [Thread-2724] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:30,567 DEBUG [Thread-2730] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f69c3f to 127.0.0.1:55165 2023-04-06 05:58:30,567 DEBUG [Thread-2730] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:30,568 DEBUG [Thread-2726] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d90797d to 127.0.0.1:55165 2023-04-06 05:58:30,568 DEBUG [Thread-2726] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:30,571 DEBUG [Thread-2722] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6523d347 to 127.0.0.1:55165 2023-04-06 05:58:30,571 DEBUG [Thread-2722] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:30,571 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/c33a7f1615e04e1781d671ff4c7be8bf 2023-04-06 05:58:30,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43214 deadline: 1680760770587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43190 deadline: 1680760770588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43202 deadline: 1680760770588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:30,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43166 deadline: 1680760770590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:30,979 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/7bd4814a455c4fa48ce804c727272342 2023-04-06 05:58:30,985 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/f45d5f46286b4403b7461bd35a25e297 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f45d5f46286b4403b7461bd35a25e297 2023-04-06 05:58:30,990 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f45d5f46286b4403b7461bd35a25e297, entries=150, sequenceid=653, filesize=11.9 K 2023-04-06 05:58:30,991 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/c33a7f1615e04e1781d671ff4c7be8bf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/c33a7f1615e04e1781d671ff4c7be8bf 2023-04-06 05:58:30,996 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/c33a7f1615e04e1781d671ff4c7be8bf, entries=150, sequenceid=653, filesize=11.9 K 2023-04-06 05:58:30,997 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/7bd4814a455c4fa48ce804c727272342 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7bd4814a455c4fa48ce804c727272342 2023-04-06 05:58:31,003 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7bd4814a455c4fa48ce804c727272342, entries=150, sequenceid=653, filesize=11.9 K 2023-04-06 05:58:31,003 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 636a24b917856079e5736bb56005308a in 1251ms, sequenceid=653, compaction requested=true 2023-04-06 05:58:31,003 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:31,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:31,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:31,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:31,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:31,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:31,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:31,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:31,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:31,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41a2d759[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:31,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:31,007 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,008 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:31,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:31,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:31,008 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,008 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:31,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:31,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:31,009 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:31,009 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:31,011 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:31,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:31,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:31,011 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:31,011 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:31,011 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,011 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:31,011 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:31,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:31,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:31,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:31,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:31,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:31,014 DEBUG [Thread-2721] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:31,014 DEBUG [Thread-2721] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:31,014 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:31,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 636a24b917856079e5736bb56005308a 2023-04-06 05:58:31,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-06 05:58:31,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:31,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:31,093 DEBUG [Thread-2717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d7deb76 to 127.0.0.1:55165 2023-04-06 05:58:31,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:31,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:31,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:31,094 DEBUG [Thread-2717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,093 DEBUG [Thread-2711] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x682c0b9e to 127.0.0.1:55165 2023-04-06 05:58:31,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:31,094 DEBUG [Thread-2711] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,096 DEBUG [Thread-2713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50d6899d to 127.0.0.1:55165 2023-04-06 05:58:31,096 DEBUG [Thread-2713] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,096 DEBUG [Thread-2719] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:55165 2023-04-06 05:58:31,096 DEBUG [Thread-2719] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,114 DEBUG [Thread-2721] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:31,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:31,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/bfa811bad99143888b4a8818518bc475 2023-04-06 05:58:31,618 DEBUG [Thread-2715] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6662e946 to 127.0.0.1:55165 2023-04-06 05:58:31,619 DEBUG [Thread-2715] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:58:31,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-06 05:58:31,619 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2466 2023-04-06 05:58:31,620 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 7397 rows 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2487 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 7461 rows 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2465 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 7394 rows 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2452 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 7354 rows 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(402): scanned 2480 2023-04-06 05:58:31,621 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(403): verified 7440 rows 2023-04-06 05:58:31,622 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:58:31,622 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a7ac4a1 to 127.0.0.1:55165 2023-04-06 05:58:31,622 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:31,624 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:58:31,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:58:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:31,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:31,630 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760711630"}]},"ts":"1680760711630"} 2023-04-06 05:58:31,631 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:58:31,636 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:58:31,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, UNASSIGN}] 2023-04-06 05:58:31,637 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, UNASSIGN 2023-04-06 05:58:31,637 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=636a24b917856079e5736bb56005308a, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:31,637 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760711637"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760711637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760711637"}]},"ts":"1680760711637"} 2023-04-06 05:58:31,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:31,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:31,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 636a24b917856079e5736bb56005308a 2023-04-06 05:58:31,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 636a24b917856079e5736bb56005308a, disabling compactions & flushes 2023-04-06 05:58:31,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:31,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/fd8db2b2ba2246148e801b3e68a6078b 2023-04-06 05:58:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:31,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/ee52cfb0d4c84ffca34009f6dabc3e14 2023-04-06 05:58:31,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/bfa811bad99143888b4a8818518bc475 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/bfa811bad99143888b4a8818518bc475 2023-04-06 05:58:31,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/bfa811bad99143888b4a8818518bc475, entries=150, sequenceid=692, filesize=11.9 K 2023-04-06 05:58:31,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/fd8db2b2ba2246148e801b3e68a6078b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fd8db2b2ba2246148e801b3e68a6078b 2023-04-06 05:58:31,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fd8db2b2ba2246148e801b3e68a6078b, entries=150, sequenceid=692, filesize=11.9 K 2023-04-06 05:58:31,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/ee52cfb0d4c84ffca34009f6dabc3e14 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ee52cfb0d4c84ffca34009f6dabc3e14 2023-04-06 05:58:31,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ee52cfb0d4c84ffca34009f6dabc3e14, entries=150, sequenceid=692, filesize=11.9 K 2023-04-06 05:58:31,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=20.13 KB/20610 for 636a24b917856079e5736bb56005308a in 870ms, sequenceid=692, compaction requested=true 2023-04-06 05:58:31,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:31,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:31,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:31,963 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. because compaction request was cancelled 2023-04-06 05:58:31,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:31,963 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. because compaction request was cancelled 2023-04-06 05:58:31,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:31,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. after waiting 0 ms 2023-04-06 05:58:31,963 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. because compaction request was cancelled 2023-04-06 05:58:31,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:31,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:31,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 636a24b917856079e5736bb56005308a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=A 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=B 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 636a24b917856079e5736bb56005308a, store=C 2023-04-06 05:58:31,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:32,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:32,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/2b3a1e83bfa64164ad0e082fb4e00454 2023-04-06 05:58:32,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:32,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/01b98205c11c4f54b54e0e81ca40039c 2023-04-06 05:58:33,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/41c93056885449d588b57bf4115d98bd 2023-04-06 05:58:33,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/A/2b3a1e83bfa64164ad0e082fb4e00454 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2b3a1e83bfa64164ad0e082fb4e00454 2023-04-06 05:58:33,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2b3a1e83bfa64164ad0e082fb4e00454, entries=100, sequenceid=698, filesize=9.5 K 2023-04-06 05:58:33,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/B/01b98205c11c4f54b54e0e81ca40039c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/01b98205c11c4f54b54e0e81ca40039c 2023-04-06 05:58:33,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/01b98205c11c4f54b54e0e81ca40039c, entries=100, sequenceid=698, filesize=9.5 K 2023-04-06 05:58:33,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/.tmp/C/41c93056885449d588b57bf4115d98bd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/41c93056885449d588b57bf4115d98bd 2023-04-06 05:58:33,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/41c93056885449d588b57bf4115d98bd, entries=100, sequenceid=698, filesize=9.5 K 2023-04-06 05:58:33,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 636a24b917856079e5736bb56005308a in 1262ms, sequenceid=698, compaction requested=true 2023-04-06 05:58:33,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4e3443211ddd4f3a905ed2ac7090eb04, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/8eb03a28e0674606b0f027906190c476, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f327de7d85e840b5870bfb244f94755c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/802e730daca64889ba0b67bf8cb1603f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/efb11da8cbcd48f79f7c110225e270fb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/798b9c7230dd44a69141e58f4d8fd31c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/9a29ff08c7454a9caa706f456c177863, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4] to archive 2023-04-06 05:58:33,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:33,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e767bb3de902486e8d51699da9b6c492 2023-04-06 05:58:33,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/0bca782557604741b1720600543f5552 2023-04-06 05:58:33,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/16fd1e1e130e48ee8fe01678babc5dbd 2023-04-06 05:58:33,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4e3443211ddd4f3a905ed2ac7090eb04 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4e3443211ddd4f3a905ed2ac7090eb04 2023-04-06 05:58:33,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d36ae741b29d43948d51e7c936513be2 2023-04-06 05:58:33,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/4b9fdab1983440ef86a6b100e6a0fdfb 2023-04-06 05:58:33,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2dd5b439e63647aa9a2d5ca5f27eef87 2023-04-06 05:58:33,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/8eb03a28e0674606b0f027906190c476 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/8eb03a28e0674606b0f027906190c476 2023-04-06 05:58:33,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/279d56e131b340e9930470f71813a4c3 2023-04-06 05:58:33,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/5ae328250e7647e3ab3bc727e45d4560 2023-04-06 05:58:33,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f327de7d85e840b5870bfb244f94755c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f327de7d85e840b5870bfb244f94755c 2023-04-06 05:58:33,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f2a6b7797d63436f87f0eb98a063e00f 2023-04-06 05:58:33,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/d716686592fb48feada17d8e7a9b3ed0 2023-04-06 05:58:33,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/802e730daca64889ba0b67bf8cb1603f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/802e730daca64889ba0b67bf8cb1603f 2023-04-06 05:58:33,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f83a3c360fdd4d0591ae16cdb6f202e9 2023-04-06 05:58:33,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/464373430f4d4819b022bb79a8409b6f 2023-04-06 05:58:33,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/efb11da8cbcd48f79f7c110225e270fb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/efb11da8cbcd48f79f7c110225e270fb 2023-04-06 05:58:33,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/3f94c0dd01ce4f61b9ca4a9462f48085 2023-04-06 05:58:33,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/775c00f4d8eb4cde9e637767b5fb4e92 2023-04-06 05:58:33,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/798b9c7230dd44a69141e58f4d8fd31c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/798b9c7230dd44a69141e58f4d8fd31c 2023-04-06 05:58:33,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/48567295f074414e87cb206a45cf9f18 2023-04-06 05:58:33,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/87ab400859de4a3c9b1ea614b550fe7b 2023-04-06 05:58:33,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/9a29ff08c7454a9caa706f456c177863 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/9a29ff08c7454a9caa706f456c177863 2023-04-06 05:58:33,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b45bfeca38e34078a284aaf6f0c85fe0 2023-04-06 05:58:33,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/65f90d8681944f9ba713aca99d2f26c4 2023-04-06 05:58:33,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/818de9bce03c40a3ade8ab7abe3cd059, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/bf15fd8dc1154d67bdf84c72b63dc932, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/686385adf2374b5dadb481cebd2b96ba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ba0d9cf7146f42d3be70b2179acde3cf, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f7534180c8eb46f0900bf381709f4cfb, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/426d28d899cd4e2dbbaf5c22d58d5e56, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/cc767558e18948b6bfb385e8e12ffa40, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114] to archive 2023-04-06 05:58:33,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:33,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ec50bc75ec5d499c8b21351f9752c8e9 2023-04-06 05:58:33,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/936b5835e6604d34b3ee873702297dca 2023-04-06 05:58:33,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/818de9bce03c40a3ade8ab7abe3cd059 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/818de9bce03c40a3ade8ab7abe3cd059 2023-04-06 05:58:33,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/1bc9c247e66e4ec6b9b88c67071387b4 2023-04-06 05:58:33,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/d2474b502f164dc2af540f3d4b952e6e 2023-04-06 05:58:33,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4ff123d82042208aad704452b45ae9 2023-04-06 05:58:33,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/bf15fd8dc1154d67bdf84c72b63dc932 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/bf15fd8dc1154d67bdf84c72b63dc932 2023-04-06 05:58:33,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/3b4943043ddf49cb996fd980d20f30c1 2023-04-06 05:58:33,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/215b5ddabc9a475a8ffe4dc964406cce 2023-04-06 05:58:33,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/686385adf2374b5dadb481cebd2b96ba to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/686385adf2374b5dadb481cebd2b96ba 2023-04-06 05:58:33,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/88589ae4cd0a4bc2816f1e61c0ea16ee 2023-04-06 05:58:33,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/4d2b19ab0217405997fae7ac7c378a78 2023-04-06 05:58:33,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ba0d9cf7146f42d3be70b2179acde3cf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/ba0d9cf7146f42d3be70b2179acde3cf 2023-04-06 05:58:33,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/066eb74ad1b347c9bc92b7ac8e8fc089 2023-04-06 05:58:33,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/653eed6351d04f76a61343d014bd7175 2023-04-06 05:58:33,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f7534180c8eb46f0900bf381709f4cfb to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f7534180c8eb46f0900bf381709f4cfb 2023-04-06 05:58:33,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5f597b155b3641cd83e44b5643609995 2023-04-06 05:58:33,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5a3fbdda59c944a3b270e5f68cb9969a 2023-04-06 05:58:33,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/426d28d899cd4e2dbbaf5c22d58d5e56 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/426d28d899cd4e2dbbaf5c22d58d5e56 2023-04-06 05:58:33,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fcca6f3efbc44428af0104269156db77 2023-04-06 05:58:33,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/29316c9497944d3cb913d2a7d77540e6 2023-04-06 05:58:33,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/cc767558e18948b6bfb385e8e12ffa40 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/cc767558e18948b6bfb385e8e12ffa40 2023-04-06 05:58:33,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/25c02284b476482c9b8e7a3234eb20c5 2023-04-06 05:58:33,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/51f6e79a90dc406abf52cb97695b0a2f 2023-04-06 05:58:33,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/f60fb885d3e24be6abd5362ad7636114 2023-04-06 05:58:33,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c9309ca0f34d49ca9b7db116d3a5398e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/935010bcb4994e1883584b759592c0d0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/6c87ee003a2346d7ba56ca9e9cfa9390, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/1fd888c2a8764f56af843bc7b2610ef1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/8f217d323d14472593463773972bf2d6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e51e589472da4052b965d2daf649cd12, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c6f31d18563444658b03a96c901b9a85, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9] to archive 2023-04-06 05:58:33,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:33,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c0738d38353645c582dcfba56aedf60a 2023-04-06 05:58:33,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2499f8678d564ea488083cd2080e64b2 2023-04-06 05:58:33,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c9309ca0f34d49ca9b7db116d3a5398e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c9309ca0f34d49ca9b7db116d3a5398e 2023-04-06 05:58:33,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c69de0dc3b374de6ac6c1c5fa68f0487 2023-04-06 05:58:33,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4afbd0a441b14815a6d188a23b512b78 2023-04-06 05:58:33,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/a0eea39427bd4c35aeca4ce201fe8363 2023-04-06 05:58:33,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/935010bcb4994e1883584b759592c0d0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/935010bcb4994e1883584b759592c0d0 2023-04-06 05:58:33,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0972670a933c4d44a1a923543b1ad3d3 2023-04-06 05:58:33,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/0d50340c3b6b4fc994e51b9e572b10dc 2023-04-06 05:58:33,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/6c87ee003a2346d7ba56ca9e9cfa9390 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/6c87ee003a2346d7ba56ca9e9cfa9390 2023-04-06 05:58:33,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/5d9a963d7ae1412ca2fd0f4cfa71f96e 2023-04-06 05:58:33,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/2bdff987de784a769b88f7416e23befc 2023-04-06 05:58:33,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/1fd888c2a8764f56af843bc7b2610ef1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/1fd888c2a8764f56af843bc7b2610ef1 2023-04-06 05:58:33,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/680a4d682a9442349783d836b6b1cc8f 2023-04-06 05:58:33,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/86dc4f0802b149d391ab160a5d51d5f2 2023-04-06 05:58:33,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/8f217d323d14472593463773972bf2d6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/8f217d323d14472593463773972bf2d6 2023-04-06 05:58:33,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/692e0249bbb24858aa3677ecc624566f 2023-04-06 05:58:33,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/4eef58263e60416eb6021b3189cfda0e 2023-04-06 05:58:33,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e51e589472da4052b965d2daf649cd12 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e51e589472da4052b965d2daf649cd12 2023-04-06 05:58:33,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7ff6c0d51672401f924d27ae77dec0c5 2023-04-06 05:58:33,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ab9d16e49519459bbe33c9eac72c4bda 2023-04-06 05:58:33,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c6f31d18563444658b03a96c901b9a85 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c6f31d18563444658b03a96c901b9a85 2023-04-06 05:58:33,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/435ac4d14630408691fec4efeb050c2b 2023-04-06 05:58:33,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/df9cb98a675246b19684cb7426484765 2023-04-06 05:58:33,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/b3e32e7bd972431fbf977da5ba9b8df9 2023-04-06 05:58:33,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/recovered.edits/701.seqid, newMaxSeqId=701, maxSeqId=1 2023-04-06 05:58:33,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a. 2023-04-06 05:58:33,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 636a24b917856079e5736bb56005308a: 2023-04-06 05:58:33,334 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 636a24b917856079e5736bb56005308a 2023-04-06 05:58:33,335 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=636a24b917856079e5736bb56005308a, regionState=CLOSED 2023-04-06 05:58:33,336 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760713335"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760713335"}]},"ts":"1680760713335"} 2023-04-06 05:58:33,339 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-06 05:58:33,339 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 636a24b917856079e5736bb56005308a, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.6990 sec 2023-04-06 05:58:33,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-06 05:58:33,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=636a24b917856079e5736bb56005308a, UNASSIGN in 1.7030 sec 2023-04-06 05:58:33,342 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760713342"}]},"ts":"1680760713342"} 2023-04-06 05:58:33,343 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:58:33,345 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:58:33,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7200 sec 2023-04-06 05:58:33,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-06 05:58:33,736 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-06 05:58:33,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:58:33,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,739 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,740 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 05:58:33,748 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:33,750 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/recovered.edits] 2023-04-06 05:58:33,753 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2b3a1e83bfa64164ad0e082fb4e00454 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/2b3a1e83bfa64164ad0e082fb4e00454 2023-04-06 05:58:33,754 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b97314e907b84e92922ab3908162c392 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/b97314e907b84e92922ab3908162c392 2023-04-06 05:58:33,755 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/bfa811bad99143888b4a8818518bc475 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/bfa811bad99143888b4a8818518bc475 2023-04-06 05:58:33,756 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e4e8917804f045b9984a99b86709056d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/e4e8917804f045b9984a99b86709056d 2023-04-06 05:58:33,757 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f45d5f46286b4403b7461bd35a25e297 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/A/f45d5f46286b4403b7461bd35a25e297 2023-04-06 05:58:33,759 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/01b98205c11c4f54b54e0e81ca40039c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/01b98205c11c4f54b54e0e81ca40039c 2023-04-06 05:58:33,759 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/5e5a1e7b25e64c0fb9e3fcefecae4fe8 2023-04-06 05:58:33,760 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/8f05b78ef21c46c6ae119779cd25b69e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/8f05b78ef21c46c6ae119779cd25b69e 2023-04-06 05:58:33,760 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/c33a7f1615e04e1781d671ff4c7be8bf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/c33a7f1615e04e1781d671ff4c7be8bf 2023-04-06 05:58:33,761 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fd8db2b2ba2246148e801b3e68a6078b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/B/fd8db2b2ba2246148e801b3e68a6078b 2023-04-06 05:58:33,762 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/41c93056885449d588b57bf4115d98bd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/41c93056885449d588b57bf4115d98bd 2023-04-06 05:58:33,763 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7bd4814a455c4fa48ce804c727272342 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/7bd4814a455c4fa48ce804c727272342 2023-04-06 05:58:33,764 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c29712cf0b5b4be5a77464c86130b187 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/c29712cf0b5b4be5a77464c86130b187 2023-04-06 05:58:33,765 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e0e25cc2d1be4d428486514b25845335 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/e0e25cc2d1be4d428486514b25845335 2023-04-06 05:58:33,766 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ee52cfb0d4c84ffca34009f6dabc3e14 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/C/ee52cfb0d4c84ffca34009f6dabc3e14 2023-04-06 05:58:33,767 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/recovered.edits/701.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a/recovered.edits/701.seqid 2023-04-06 05:58:33,768 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/636a24b917856079e5736bb56005308a 2023-04-06 05:58:33,768 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:58:33,769 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,770 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:58:33,772 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:58:33,772 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,772 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:58:33,772 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760713772"}]},"ts":"9223372036854775807"} 2023-04-06 05:58:33,774 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:58:33,774 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 636a24b917856079e5736bb56005308a, NAME => 'TestAcidGuarantees,,1680760689348.636a24b917856079e5736bb56005308a.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:58:33,774 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:58:33,774 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760713774"}]},"ts":"9223372036854775807"} 2023-04-06 05:58:33,775 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:58:33,776 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,777 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 39 msec 2023-04-06 05:58:33,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-06 05:58:33,842 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-06 05:58:33,857 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 227) - Thread LEAK? -, OpenFileDescriptor=538 (was 552), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=421 (was 446), ProcessCount=167 (was 167), AvailableMemoryMB=4284 (was 4282) - AvailableMemoryMB LEAK? - 2023-04-06 05:58:33,869 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=538, MaxFileDescriptor=60000, SystemLoadAverage=421, ProcessCount=167, AvailableMemoryMB=4283 2023-04-06 05:58:33,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:58:33,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:58:33,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:33,873 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-06 05:58:33,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-06 05:58:33,873 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-06 05:58:33,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 05:58:33,875 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:33,875 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 empty. 2023-04-06 05:58:33,875 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:33,876 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:58:33,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 05:58:34,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 05:58:34,284 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:58:34,287 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0938af50bcbc9322cb599bce38356083, NAME => 'TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0938af50bcbc9322cb599bce38356083, disabling compactions & flushes 2023-04-06 05:58:34,296 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. after waiting 0 ms 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,296 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,296 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:34,299 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-06 05:58:34,300 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760714300"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760714300"}]},"ts":"1680760714300"} 2023-04-06 05:58:34,302 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-06 05:58:34,302 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-06 05:58:34,302 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760714302"}]},"ts":"1680760714302"} 2023-04-06 05:58:34,303 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-06 05:58:34,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, ASSIGN}] 2023-04-06 05:58:34,307 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, ASSIGN 2023-04-06 05:58:34,307 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=false 2023-04-06 05:58:34,460 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:34,460 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760714459"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760714459"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760714459"}]},"ts":"1680760714459"} 2023-04-06 05:58:34,462 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:34,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 05:58:34,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0938af50bcbc9322cb599bce38356083, NAME => 'TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:58:34,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:58:34,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,628 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,629 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A 2023-04-06 05:58:34,630 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A 2023-04-06 05:58:34,630 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:34,630 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName A 2023-04-06 05:58:34,631 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:34,631 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,632 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B 2023-04-06 05:58:34,633 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B 2023-04-06 05:58:34,633 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:34,633 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName B 2023-04-06 05:58:34,634 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:34,634 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,635 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C 2023-04-06 05:58:34,635 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C 2023-04-06 05:58:34,636 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:34,636 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName C 2023-04-06 05:58:34,637 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:34,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,641 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:58:34,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:34,647 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-06 05:58:34,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0938af50bcbc9322cb599bce38356083; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63232943, jitterRate=-0.0577557235956192}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:58:34,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:34,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., pid=56, masterSystemTime=1680760714614 2023-04-06 05:58:34,650 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:34,651 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:34,651 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760714651"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760714651"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760714651"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760714651"}]},"ts":"1680760714651"} 2023-04-06 05:58:34,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-06 05:58:34,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 in 191 msec 2023-04-06 05:58:34,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-06 05:58:34,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, ASSIGN in 350 msec 2023-04-06 05:58:34,657 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-06 05:58:34,657 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760714657"}]},"ts":"1680760714657"} 2023-04-06 05:58:34,658 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-06 05:58:34,660 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-06 05:58:34,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 789 msec 2023-04-06 05:58:34,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-06 05:58:34,980 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-06 05:58:34,980 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x6722b207 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:34,985 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f11e84a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:34,986 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:34,987 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:34,988 DEBUG [Listener at localhost.localdomain/33939] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-06 05:58:34,990 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-06 05:58:34,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-06 05:58:34,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-06 05:58:34,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:35,404 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:58:35,407 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-06 05:58:35,408 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-06 05:58:35,412 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-06 05:58:35,415 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, REOPEN/MOVE}] 2023-04-06 05:58:35,416 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, REOPEN/MOVE 2023-04-06 05:58:35,416 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:35,417 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760715416"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760715416"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760715416"}]},"ts":"1680760715416"} 2023-04-06 05:58:35,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:35,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0938af50bcbc9322cb599bce38356083, disabling compactions & flushes 2023-04-06 05:58:35,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. after waiting 0 ms 2023-04-06 05:58:35,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-06 05:58:35,577 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,577 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:35,577 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 0938af50bcbc9322cb599bce38356083 to self. 2023-04-06 05:58:35,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,581 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=CLOSED 2023-04-06 05:58:35,581 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760715581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760715581"}]},"ts":"1680760715581"} 2023-04-06 05:58:35,585 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-06 05:58:35,585 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 in 165 msec 2023-04-06 05:58:35,586 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,33239,1680760570949; forceNewPlan=false, retain=true 2023-04-06 05:58:35,737 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:35,737 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760715737"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760715737"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760715737"}]},"ts":"1680760715737"} 2023-04-06 05:58:35,740 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:35,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0938af50bcbc9322cb599bce38356083, NAME => 'TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.', STARTKEY => '', ENDKEY => ''} 2023-04-06 05:58:35,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-06 05:58:35,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,902 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,903 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A 2023-04-06 05:58:35,903 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A 2023-04-06 05:58:35,904 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:35,904 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName A 2023-04-06 05:58:35,906 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:35,906 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,907 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B 2023-04-06 05:58:35,907 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B 2023-04-06 05:58:35,907 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:35,908 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName B 2023-04-06 05:58:35,908 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:35,908 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,909 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C 2023-04-06 05:58:35,910 DEBUG [StoreOpener-0938af50bcbc9322cb599bce38356083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C 2023-04-06 05:58:35,910 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-06 05:58:35,910 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0938af50bcbc9322cb599bce38356083 columnFamilyName C 2023-04-06 05:58:35,911 INFO [StoreOpener-0938af50bcbc9322cb599bce38356083-1] regionserver.HStore(310): Store=0938af50bcbc9322cb599bce38356083/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-06 05:58:35,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,915 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-06 05:58:35,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:35,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 0938af50bcbc9322cb599bce38356083; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58955288, jitterRate=-0.1214977502822876}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-06 05:58:35,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:35,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., pid=61, masterSystemTime=1680760715892 2023-04-06 05:58:35,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:35,920 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:35,920 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760715920"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680760715920"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680760715920"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680760715920"}]},"ts":"1680760715920"} 2023-04-06 05:58:35,922 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-06 05:58:35,923 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 in 181 msec 2023-04-06 05:58:35,924 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-06 05:58:35,924 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, REOPEN/MOVE in 508 msec 2023-04-06 05:58:35,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-06 05:58:35,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-04-06 05:58:35,930 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 936 msec 2023-04-06 05:58:35,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-06 05:58:35,933 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x48502121 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,938 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a1010a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,938 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x08d3d8bc to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,944 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4143b9ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,945 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7e6ad437 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,948 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@686af57e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,948 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x2f6f7187 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,952 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78cd331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,953 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x49c34fbe to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,958 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d74fc48, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,958 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x070ab5f5 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,962 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3da68fe0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,963 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x2daeb795 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,975 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fd90ac8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,975 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x28ac2b09 to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,983 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e63396, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,984 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x7bf1c01c to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:35,992 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32cc4808, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:35,992 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(139): Connect 0x43591f8a to 127.0.0.1:55165 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-06 05:58:36,000 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36ddd6dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-06 05:58:36,011 DEBUG [hconnection-0x10944f81-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,011 DEBUG [hconnection-0x1a97d513-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,011 DEBUG [hconnection-0x49f70225-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,011 DEBUG [hconnection-0x7de9b5f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,015 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44514, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,015 DEBUG [hconnection-0x671acf3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,016 DEBUG [hconnection-0x5ad0580a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,016 DEBUG [hconnection-0x605240c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,016 DEBUG [hconnection-0x46213cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,016 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,017 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,019 DEBUG [hconnection-0x4c34c5f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,019 DEBUG [hconnection-0x4b23fe02-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-06 05:58:36,024 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,024 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,024 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,024 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,025 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,025 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,027 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-06 05:58:36,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,035 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,036 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,036 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:36,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:36,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:36,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,044 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,046 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,047 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,049 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,049 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,050 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eeb0b42[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,051 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,052 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,052 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,053 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,053 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,055 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,055 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,055 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760776053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760776054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,056 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760776053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760776054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760776054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,056 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,057 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,059 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,059 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,059 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,059 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760776156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760776156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760776157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,160 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760776159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760776159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,162 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,162 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,166 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,166 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,171 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,174 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,175 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,175 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,176 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,177 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,177 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,178 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4482d7f9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,180 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,183 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,183 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,187 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,187 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,287 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,290 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,290 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,291 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,291 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,292 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,292 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,293 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,293 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,293 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,293 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,293 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,294 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,294 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b7e0c63[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,296 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,296 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,296 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,297 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,297 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,300 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,300 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,300 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,300 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,300 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,301 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,301 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,301 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,304 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,304 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,304 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760776358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760776359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760776359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760776362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760776362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,405 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,407 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,407 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,408 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,408 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,409 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,409 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,409 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,410 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,410 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,410 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,410 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c311c4a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,412 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,415 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,415 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,416 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,417 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,418 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,418 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,418 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,418 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,419 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,454 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304061b40786055ed4220a163a0a915c75234 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b40786055ed4220a163a0a915c75234 2023-04-06 05:58:36,519 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,521 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,521 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,522 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,522 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,523 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,523 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,523 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,523 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,524 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,524 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,524 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,525 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa6ab95[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,525 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,526 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,526 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,527 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,529 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,529 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,530 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,531 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,533 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,533 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,633 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,636 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,636 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,637 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,637 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,638 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,638 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,638 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,638 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,638 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,639 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,639 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,640 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c99027[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,641 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,642 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,642 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,643 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,643 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,644 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,644 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,644 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,646 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,649 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,649 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760776660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760776660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760776660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760776663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:36,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760776666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,749 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,751 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,751 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,752 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,752 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,753 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,753 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,753 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,754 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,754 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,754 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,755 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,755 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539f8a49[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,756 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,757 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,758 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,760 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,760 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,760 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,761 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,764 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,764 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,765 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,765 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,859 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/c9d74e3083f045cab594882a44af91c2 2023-04-06 05:58:36,865 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:36,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:36,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:36,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:36,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:36,868 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:36,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:36,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:36,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:36,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:36,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:36,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:36,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:36,871 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:36,871 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:36,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,872 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,872 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,872 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:36,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37280f3c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:36,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:36,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,891 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:36,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:36,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:36,891 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:36,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:36,894 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,894 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:36,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:36,896 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:36,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:36,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:36,896 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:36,896 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:36,897 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,897 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,897 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:36,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:36,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,899 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:36,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:36,901 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:36,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:36,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:36,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:36,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:36,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:36,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:36,901 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:36,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:36,902 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,002 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,004 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,005 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,005 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,006 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,006 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,006 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,006 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,007 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,007 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,007 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:37,007 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,007 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,008 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,008 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482de04f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,009 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,009 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,009 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,010 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,010 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,010 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,012 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,012 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,013 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,015 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,017 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,017 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,017 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,117 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,119 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,120 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,120 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,121 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,121 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,121 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,121 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,122 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,122 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,122 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:37,122 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,122 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,123 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,123 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,123 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,123 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb109c5[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,124 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,125 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,125 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,141 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,141 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,143 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,143 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,144 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,145 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,146 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,147 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,147 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760777163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:37,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760777163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:37,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760777164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760777167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:37,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760777170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,247 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,250 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,250 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,251 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,252 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,252 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,252 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:37,252 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,253 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,254 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@335315d8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,255 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,255 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,256 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,256 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,259 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,259 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,259 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/d6446c81112541de91c30fe7af83fa87 2023-04-06 05:58:37,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,269 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,269 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,270 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,270 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/ec2f551a6c4f46d59cffb04536bbde8f 2023-04-06 05:58:37,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/c9d74e3083f045cab594882a44af91c2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2 2023-04-06 05:58:37,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2, entries=150, sequenceid=16, filesize=25.1 K 2023-04-06 05:58:37,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/d6446c81112541de91c30fe7af83fa87 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87 2023-04-06 05:58:37,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 05:58:37,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/ec2f551a6c4f46d59cffb04536bbde8f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f 2023-04-06 05:58:37,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f, entries=150, sequenceid=16, filesize=11.6 K 2023-04-06 05:58:37,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0938af50bcbc9322cb599bce38356083 in 1255ms, sequenceid=16, compaction requested=false 2023-04-06 05:58:37,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,370 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,373 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,374 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,375 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,375 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,375 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,375 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:58:37,375 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:37,376 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:37,376 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:37,376 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:37,376 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:37,376 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:37,395 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406430b8f385f934151b93bbccbb1a8f4b3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406430b8f385f934151b93bbccbb1a8f4b3 2023-04-06 05:58:37,400 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/ae5ab87bc788400d93a1e4b357690c95 2023-04-06 05:58:37,409 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/425f018178ed4530aea9a38b1bd4656b 2023-04-06 05:58:37,424 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/7f53666d64fd4b19acde65962af3bb26 2023-04-06 05:58:37,428 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/ae5ab87bc788400d93a1e4b357690c95 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95 2023-04-06 05:58:37,432 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95, entries=150, sequenceid=36, filesize=25.1 K 2023-04-06 05:58:37,433 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/425f018178ed4530aea9a38b1bd4656b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b 2023-04-06 05:58:37,436 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 05:58:37,436 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/7f53666d64fd4b19acde65962af3bb26 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26 2023-04-06 05:58:37,439 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26, entries=150, sequenceid=36, filesize=11.6 K 2023-04-06 05:58:37,440 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 0938af50bcbc9322cb599bce38356083 in 65ms, sequenceid=36, compaction requested=false 2023-04-06 05:58:37,440 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,440 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,440 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,440 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,440 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,440 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,440 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f5bd61[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,443 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,443 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,444 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,446 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,446 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,447 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,447 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,447 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,449 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,451 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,451 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,451 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,451 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,551 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,554 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,554 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,555 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,555 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,558 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,558 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,558 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,559 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,559 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,560 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,560 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63fd0228[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,562 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,562 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,563 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,564 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,566 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,566 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,568 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,571 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,571 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,571 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,671 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,674 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,675 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,676 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,676 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,677 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,677 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,677 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,677 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,677 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,678 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7716d0b3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,680 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,680 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,683 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,684 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,685 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,687 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,687 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,688 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,788 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,792 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,793 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,794 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,794 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,794 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:37,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,796 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,797 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,797 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc0f65c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,799 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,800 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,802 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,802 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,802 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,802 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,802 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,804 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,805 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,806 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,807 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,807 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,907 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:37,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:37,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:37,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:37,910 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:37,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:37,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:37,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,912 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:37,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:37,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:37,913 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,913 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:37,913 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:37,913 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:37,913 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:37,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:37,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,914 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:37,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35375610[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:37,916 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:37,916 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,916 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:37,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:37,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:37,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:37,919 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:37,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:37,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:37,920 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:37,920 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:37,920 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,920 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,921 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:37,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:37,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:37,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:37,924 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:37,925 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:37,925 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:37,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,025 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,028 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,030 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,030 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,030 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:38,030 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,031 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,032 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e50eedc[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,034 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,034 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,034 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,034 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,034 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,035 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,037 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,039 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,039 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,040 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,041 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,042 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,043 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,043 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,044 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,044 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,044 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,144 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,147 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,148 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,149 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,149 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,149 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,149 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,149 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,149 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,149 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,149 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:38,149 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,150 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,152 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,154 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0232ed[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,154 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,154 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,155 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,156 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,156 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,156 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,157 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,158 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,159 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,159 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,160 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,160 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,161 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,163 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,163 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,163 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,164 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:38,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:38,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760718200, max timestamp=1680760718209Num uniques 859; 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760718200, max timestamp=1680760718209Num uniques 859; 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760718200, max timestamp=1680760718209Num uniques 859; 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:38,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:38,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760778221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760778222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760778222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760778223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760778224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,264 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,266 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,268 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,268 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,268 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,268 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,268 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,269 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,269 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,269 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,269 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,269 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,270 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@242b4947[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,271 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,271 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,272 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,272 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,272 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,275 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,275 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,275 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,275 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,275 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,275 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,276 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,277 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,278 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,278 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,279 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,279 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760778325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760778325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760778325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760778327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760778327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,379 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,381 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,382 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,384 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,384 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,384 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,384 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,384 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,385 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,385 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba30c04[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,387 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,388 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,390 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,390 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,390 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,390 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,391 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,392 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,393 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,393 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,393 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,393 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,493 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,496 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,497 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,497 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,498 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,498 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,498 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,498 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,498 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,499 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,499 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,499 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,499 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,499 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,500 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@123c31a2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,502 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,505 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,505 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,505 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,505 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,505 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,506 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,509 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,509 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,509 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,510 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,510 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760778528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760778528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760778528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760778529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760778529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,599 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065fdef8e0eaf5471e8e079db23c133408 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fdef8e0eaf5471e8e079db23c133408 2023-04-06 05:58:38,603 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=53, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/f43363bcd21c4706a0b5aa830f3353ac 2023-04-06 05:58:38,610 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,612 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,613 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,613 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,614 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,615 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,615 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,615 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,615 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,615 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,616 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c47fc52[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,617 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,618 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,618 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,620 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,620 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,620 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,621 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,621 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,621 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,621 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,622 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,623 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,623 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,624 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,624 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,624 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,724 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,727 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,729 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,729 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,729 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,729 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,729 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8f3ec[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,732 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,732 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,732 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,732 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,733 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,734 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,734 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,735 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,735 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,735 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,735 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,736 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,738 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,738 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,739 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,743 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760778830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760778832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760778832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760778833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760778833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,839 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,841 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,843 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,843 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,844 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31a81c9f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,846 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,847 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,847 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,847 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,847 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,849 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,849 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,849 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,850 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,850 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,851 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,853 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,853 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,853 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,853 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,858 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-06 05:58:38,954 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:38,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:38,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:38,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:38,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:38,956 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:38,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:38,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:38,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:38,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:38,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:38,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:38,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:38,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:38,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:38,958 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:38,958 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,958 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:38,958 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:38,959 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:38,959 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,960 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,960 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,960 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:38,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ea9a29[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:38,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:38,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:38,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:38,961 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:38,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,962 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:38,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,962 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,963 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,963 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:38,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:38,964 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:38,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:38,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:38,964 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:38,964 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:38,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,965 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:38,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,966 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:38,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,967 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:38,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:38,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:38,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:38,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:38,969 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:38,969 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:38,969 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:38,969 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:38,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/260cb60ec84c442eba620fa6c9f2cd46 2023-04-06 05:58:39,069 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:39,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:39,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:39,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:39,072 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:39,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:39,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:39,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:39,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:39,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:39,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:39,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:39,074 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:39,075 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:39,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,076 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,076 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,076 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:39,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c5b4a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:39,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:39,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:39,077 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:39,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,078 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:39,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:39,078 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:39,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,078 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:39,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:39,080 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:39,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:39,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:39,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:39,080 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:39,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,081 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,082 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,083 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:39,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:39,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,084 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,084 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,085 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:39,085 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:39,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:39,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,085 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,185 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:39,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:39,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:39,187 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:39,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:39,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:39,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:39,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:39,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:39,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:39,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:39,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:39,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:39,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@440e9f04[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:39,201 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:39,201 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,207 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:39,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:39,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:39,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,208 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:39,210 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,210 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:39,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:39,212 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:39,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:39,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:39,212 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:39,212 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:39,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,213 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:39,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:39,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:39,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,217 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:39,216 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,217 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:39,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,317 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:39,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:39,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:39,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:39,319 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:39,319 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,319 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:39,319 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:39,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,320 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,320 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:39,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:39,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:39,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:39,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:39,321 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:39,321 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,322 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:39,322 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:39,322 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:39,322 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,322 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:39,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b33ffe[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:39,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:39,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,324 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:39,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,324 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:39,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:39,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:39,325 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:39,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:39,327 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:39,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:39,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:39,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:39,327 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:39,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,328 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:39,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:39,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:39,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,331 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:39,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:39,331 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:39,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:39,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760779334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:39,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760779335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:39,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760779335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:39,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760779337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:39,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760779339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=53 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/a35b8af07ff54aabb71d149322ac0d6c 2023-04-06 05:58:39,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/f43363bcd21c4706a0b5aa830f3353ac as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac 2023-04-06 05:58:39,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac, entries=250, sequenceid=53, filesize=38.7 K 2023-04-06 05:58:39,425 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/260cb60ec84c442eba620fa6c9f2cd46 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46 2023-04-06 05:58:39,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46, entries=150, sequenceid=53, filesize=11.6 K 2023-04-06 05:58:39,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/a35b8af07ff54aabb71d149322ac0d6c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c 2023-04-06 05:58:39,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c, entries=150, sequenceid=53, filesize=11.6 K 2023-04-06 05:58:39,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0938af50bcbc9322cb599bce38356083 in 1243ms, sequenceid=53, compaction requested=true 2023-04-06 05:58:39,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:39,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:39,431 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:39,431 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:39,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:39,431 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:39,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:39,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:39,432 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,432 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,432 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=89.0 K 2023-04-06 05:58:39,432 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=34.8 K 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c9d74e3083f045cab594882a44af91c2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680760716038 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d6446c81112541de91c30fe7af83fa87, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680760716038 2023-04-06 05:58:39,432 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ae5ab87bc788400d93a1e4b357690c95, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680760716053 2023-04-06 05:58:39,433 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 425f018178ed4530aea9a38b1bd4656b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680760716053 2023-04-06 05:58:39,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:39,433 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f43363bcd21c4706a0b5aa830f3353ac, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:39,433 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 260cb60ec84c442eba620fa6c9f2cd46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:39,434 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:39,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:39,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:39,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,434 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,435 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:39,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:39,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:39,435 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:39,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:39,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:39,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:39,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:39,436 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:39,436 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,436 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:39,437 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:39,437 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:39,438 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#439 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:39,441 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#438 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:39,465 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304065fab0a5776214c0c815aec303164881d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fab0a5776214c0c815aec303164881d 2023-04-06 05:58:39,475 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/d8f5d26b5ea143b08718d917478aa4b9 2023-04-06 05:58:39,850 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/29ea0e733cce4a06b8769590de1d6d32 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29ea0e733cce4a06b8769590de1d6d32 2023-04-06 05:58:39,853 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 29ea0e733cce4a06b8769590de1d6d32(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:39,854 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:39,854 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760719431; duration=0sec 2023-04-06 05:58:39,854 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:39,854 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:39,854 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:39,855 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:39,855 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:39,855 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=34.8 K 2023-04-06 05:58:39,855 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ec2f551a6c4f46d59cffb04536bbde8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680760716038 2023-04-06 05:58:39,855 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7f53666d64fd4b19acde65962af3bb26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680760716053 2023-04-06 05:58:39,855 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a35b8af07ff54aabb71d149322ac0d6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:39,860 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#442 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:39,888 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/0ab9aa3e68764d30a0939c35c036bb0d 2023-04-06 05:58:40,266 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/0aae6df55f764b989cc56f620e9b3bec as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0aae6df55f764b989cc56f620e9b3bec 2023-04-06 05:58:40,269 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 0aae6df55f764b989cc56f620e9b3bec(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:40,269 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:40,269 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760719431; duration=0sec 2023-04-06 05:58:40,269 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:40,270 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/2fee635788084055a39b9287a990d9f1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/2fee635788084055a39b9287a990d9f1 2023-04-06 05:58:40,273 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 2fee635788084055a39b9287a990d9f1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:40,273 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:40,273 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760719431; duration=0sec 2023-04-06 05:58:40,273 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:40,295 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/163c4c6982934b9281d39675b3e41403 2023-04-06 05:58:40,298 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/d8f5d26b5ea143b08718d917478aa4b9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9 2023-04-06 05:58:40,301 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9, entries=150, sequenceid=89, filesize=25.1 K 2023-04-06 05:58:40,302 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/0ab9aa3e68764d30a0939c35c036bb0d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d 2023-04-06 05:58:40,305 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d, entries=150, sequenceid=89, filesize=11.6 K 2023-04-06 05:58:40,305 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/163c4c6982934b9281d39675b3e41403 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403 2023-04-06 05:58:40,327 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403, entries=150, sequenceid=89, filesize=11.6 K 2023-04-06 05:58:40,328 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 0938af50bcbc9322cb599bce38356083 in 891ms, sequenceid=89, compaction requested=false 2023-04-06 05:58:40,328 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:40,328 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,328 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,330 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@239ec0b3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,332 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,335 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,335 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,337 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,339 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,339 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,339 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,340 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,340 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:40,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:40,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:40,357 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:40,357 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:40,357 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:40,357 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:40,357 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:40,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760720374, max timestamp=1680760720378Num uniques 859; 2023-04-06 05:58:40,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:40,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760720374, max timestamp=1680760720378Num uniques 859; 2023-04-06 05:58:40,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:40,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760720374, max timestamp=1680760720378Num uniques 859; 2023-04-06 05:58:40,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:40,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760780393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760780392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760780393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760780398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760780398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,439 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:40,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:40,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:40,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:40,442 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:40,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:40,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:40,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,443 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:40,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:40,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:40,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:40,444 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,445 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:40,446 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,446 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:40,446 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:40,446 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,446 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,447 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,447 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,447 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402b8981[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,448 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,449 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,449 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,449 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,452 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,452 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,455 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,457 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,457 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,457 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760780499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760780499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760780499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760780503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760780503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,557 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:40,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:40,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:40,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:40,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:40,559 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:40,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:40,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:40,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,561 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,561 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:40,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:40,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:40,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:40,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:40,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:40,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:40,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:40,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:40,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:40,563 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:40,564 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70edc83f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,566 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,567 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,568 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,568 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,570 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,570 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,570 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,572 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,573 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,575 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,575 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,575 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,575 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,576 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,576 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,675 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:40,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:40,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:40,678 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:40,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:40,678 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:40,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,679 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,679 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:40,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:40,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:40,681 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,681 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:40,681 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:40,681 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,682 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8d4513[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,685 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,685 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,685 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,685 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,686 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,689 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,691 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,691 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,691 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,691 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,691 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,692 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,692 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,693 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,694 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,694 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,694 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,695 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,695 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,695 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,695 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760780701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760780701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760780702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760780705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:40,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760780706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,780 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304060ec0420f6d5943bfb228b55a5fd684c0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ec0420f6d5943bfb228b55a5fd684c0 2023-04-06 05:58:40,795 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:40,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:40,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:40,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:40,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:40,797 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,798 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:40,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:40,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:40,800 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,800 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:40,800 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:40,800 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,800 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,801 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9438d69[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,802 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,802 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,803 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,803 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,803 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,803 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,803 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,804 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,804 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,804 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,804 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,806 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,806 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,806 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,807 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,808 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,809 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,809 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,809 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,810 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:40,910 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:40,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:40,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:40,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:40,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:40,912 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:40,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:40,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:40,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:40,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:40,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:40,915 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,915 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:40,915 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:40,915 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:40,915 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:40,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,917 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15386a41[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:40,918 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:40,918 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,919 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:40,919 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,919 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:40,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:40,921 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:40,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:40,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:40,921 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:40,921 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:40,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:40,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:40,922 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,923 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:40,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:40,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:40,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:40,924 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:40,924 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:40,924 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:40,924 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:40,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:40,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:40,925 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:40,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:41,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760781005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760781005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760781006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760781007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760781011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,025 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,027 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,028 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,028 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,029 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,029 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,029 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,029 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,029 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,030 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,030 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,031 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b8af4e9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,032 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,032 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,033 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,033 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,033 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,035 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,035 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,035 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,036 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,036 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,036 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,037 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,039 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:41,039 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,039 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,039 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,139 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,142 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,144 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d198d56[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,146 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,147 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,147 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,147 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,148 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,148 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,148 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,148 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,150 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,150 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,150 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,151 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,151 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,152 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,154 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,154 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,154 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,154 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,185 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/a9a1be46c84744aaa5a864dab7cc9c21 2023-04-06 05:58:41,254 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,257 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,258 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,258 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,258 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,259 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,259 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,260 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,260 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,260 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,261 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28246759[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,262 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,262 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,263 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,263 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,264 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,264 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,266 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,266 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,266 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,267 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,267 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,268 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,270 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,270 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,270 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,270 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,372 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,375 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,375 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,376 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,376 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,377 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,377 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,377 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,377 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,377 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,378 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,378 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e986a7e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,379 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,379 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,379 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,380 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,380 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,380 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,382 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,382 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,382 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,383 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,383 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,384 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,387 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,387 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,387 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,387 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,487 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,491 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,491 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,492 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,492 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,493 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,493 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,493 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,493 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,493 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,494 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,494 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22e9e4f8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,496 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,496 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,496 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,497 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,497 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,499 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,499 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,499 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,500 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,500 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,501 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:41,502 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,502 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,503 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,503 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,503 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760781508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760781508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760781510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760781510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:41,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760781513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/85ee23875ee74abb8ee5f32303684a90 2023-04-06 05:58:41,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/d04ada10781d47c8abe706b1bd2d809f 2023-04-06 05:58:41,603 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/a9a1be46c84744aaa5a864dab7cc9c21 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21 2023-04-06 05:58:41,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,606 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,606 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,607 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,607 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21, entries=350, sequenceid=102, filesize=52.3 K 2023-04-06 05:58:41,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/85ee23875ee74abb8ee5f32303684a90 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90 2023-04-06 05:58:41,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,608 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,608 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,609 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,609 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:41,609 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:41,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:41,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:41,609 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,610 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:41,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536bbece[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:41,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:41,611 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90, entries=150, sequenceid=102, filesize=11.6 K 2023-04-06 05:58:41,612 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/d04ada10781d47c8abe706b1bd2d809f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f 2023-04-06 05:58:41,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,612 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:41,612 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:41,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:41,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,613 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:41,613 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f, entries=150, sequenceid=102, filesize=11.6 K 2023-04-06 05:58:41,614 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-06 05:58:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:41,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:41,615 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:41,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:41,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:41,615 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:41,615 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:41,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0938af50bcbc9322cb599bce38356083 in 1259ms, sequenceid=102, compaction requested=true 2023-04-06 05:58:41,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:41,615 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:41,615 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:41,616 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105105 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:41,616 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:41,616 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:41,616 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,616 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:41,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,616 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,616 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0aae6df55f764b989cc56f620e9b3bec, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=102.6 K 2023-04-06 05:58:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,616 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,617 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29ea0e733cce4a06b8769590de1d6d32, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=34.9 K 2023-04-06 05:58:41,617 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:41,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,617 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 0aae6df55f764b989cc56f620e9b3bec, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:41,617 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 29ea0e733cce4a06b8769590de1d6d32, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:41,617 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:41,617 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d8f5d26b5ea143b08718d917478aa4b9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680760718221 2023-04-06 05:58:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,617 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 0ab9aa3e68764d30a0939c35c036bb0d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680760718221 2023-04-06 05:58:41,617 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a9a1be46c84744aaa5a864dab7cc9c21, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720347 2023-04-06 05:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,618 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 85ee23875ee74abb8ee5f32303684a90, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720350 2023-04-06 05:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,618 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:41,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:41,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,624 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#448 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:41,624 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#447 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:41,628 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,628 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:41,627 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:41,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:41,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:41,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:41,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:41,628 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:41,628 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:41,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:41,728 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:41,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:41,731 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:41,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:41,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:41,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,731 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,732 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,732 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:41,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:41,732 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:41,732 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:41,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:41,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:41,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:41,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:41,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:41,733 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:41,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:41,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:41,740 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:41,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:41,740 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-06 05:58:41,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:41,740 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:41,741 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:41,741 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:41,741 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:41,741 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:42,034 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/433061bbfec14e02b822921d2901a759 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/433061bbfec14e02b822921d2901a759 2023-04-06 05:58:42,037 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 433061bbfec14e02b822921d2901a759(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:42,037 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:42,037 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760721615; duration=0sec 2023-04-06 05:58:42,038 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:42,038 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:42,038 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:42,038 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:42,038 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:42,038 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/2fee635788084055a39b9287a990d9f1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=34.9 K 2023-04-06 05:58:42,039 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 2fee635788084055a39b9287a990d9f1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=53, earliestPutTs=1680760718183 2023-04-06 05:58:42,039 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 163c4c6982934b9281d39675b3e41403, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680760718221 2023-04-06 05:58:42,039 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d04ada10781d47c8abe706b1bd2d809f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720350 2023-04-06 05:58:42,044 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:42,052 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/2ff529d2a9b44dddbecdb3991b4c5aa8 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2ff529d2a9b44dddbecdb3991b4c5aa8 2023-04-06 05:58:42,053 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/ee633458de3746f78006b1aaa6b2193c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ee633458de3746f78006b1aaa6b2193c 2023-04-06 05:58:42,056 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 2ff529d2a9b44dddbecdb3991b4c5aa8(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:42,056 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:42,056 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760721615; duration=0sec 2023-04-06 05:58:42,056 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:42,058 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into ee633458de3746f78006b1aaa6b2193c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:42,058 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:42,058 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760721615; duration=0sec 2023-04-06 05:58:42,058 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:42,161 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406b7e6757a72254dcdb6711f01fada54b3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b7e6757a72254dcdb6711f01fada54b3 2023-04-06 05:58:42,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:42,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:42,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760782519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760782519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760782521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760782522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760782522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,565 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=138, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/7bf96660271c4928aaf34210668b4685 2023-04-06 05:58:42,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760782622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760782625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760782625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760782824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760782826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:42,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760782827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:42,973 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/549668857ace437fa1b478c9d976d564 2023-04-06 05:58:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760783128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760783128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760783130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,381 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/36f575f7abe74dfda466bd28c3a8f4e9 2023-04-06 05:58:43,384 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/7bf96660271c4928aaf34210668b4685 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685 2023-04-06 05:58:43,387 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685, entries=150, sequenceid=138, filesize=25.3 K 2023-04-06 05:58:43,387 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/549668857ace437fa1b478c9d976d564 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564 2023-04-06 05:58:43,390 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564, entries=150, sequenceid=138, filesize=11.8 K 2023-04-06 05:58:43,390 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/36f575f7abe74dfda466bd28c3a8f4e9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9 2023-04-06 05:58:43,393 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9, entries=150, sequenceid=138, filesize=11.8 K 2023-04-06 05:58:43,393 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0938af50bcbc9322cb599bce38356083 in 1653ms, sequenceid=138, compaction requested=false 2023-04-06 05:58:43,393 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:43,393 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:43,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:43,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:43,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,393 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:43,394 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:43,394 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,394 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,395 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,395 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:43,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@587e0555[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:43,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:43,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,396 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:43,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:43,396 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:43,396 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,397 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:43,397 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:43,397 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:43,397 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:43,397 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:43,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:43,399 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:43,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:43,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:43,399 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:43,400 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:43,401 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,401 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:43,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:43,401 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,403 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:43,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:43,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:43,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:43,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:43,404 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,404 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:43,404 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:43,404 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:43,405 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,405 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,405 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,505 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:43,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:43,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:43,507 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:43,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:43,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:43,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:43,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,507 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,508 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:43,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,508 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:43,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:43,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:43,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:43,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:43,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:43,509 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:43,509 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:43,509 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:43,509 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:43,509 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:43,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:43,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:43,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:43,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:43,510 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:43,517 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304064e7ceca4f94142889b37d6ef3e031fe2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e7ceca4f94142889b37d6ef3e031fe2 2023-04-06 05:58:43,521 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/ceaf9e7ebf5d417b8edb1348ad01186f 2023-04-06 05:58:43,528 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/ffeffc78891742f0a5a5870ab6f105a1 2023-04-06 05:58:43,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:43,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:43,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760723641, max timestamp=1680760723650Num uniques 859; 2023-04-06 05:58:43,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760723641, max timestamp=1680760723650Num uniques 859; 2023-04-06 05:58:43,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:43,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:43,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760723641, max timestamp=1680760723650Num uniques 859; 2023-04-06 05:58:43,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:43,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760783667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760783670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760783670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760783771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760783774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760783775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,934 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/109feeb580084a1797d88c100deda0cc 2023-04-06 05:58:43,937 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/ceaf9e7ebf5d417b8edb1348ad01186f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f 2023-04-06 05:58:43,940 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f, entries=150, sequenceid=151, filesize=25.3 K 2023-04-06 05:58:43,941 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/ffeffc78891742f0a5a5870ab6f105a1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1 2023-04-06 05:58:43,944 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1, entries=150, sequenceid=151, filesize=11.8 K 2023-04-06 05:58:43,945 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/109feeb580084a1797d88c100deda0cc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc 2023-04-06 05:58:43,948 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc, entries=150, sequenceid=151, filesize=11.8 K 2023-04-06 05:58:43,948 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 0938af50bcbc9322cb599bce38356083 in 439ms, sequenceid=151, compaction requested=true 2023-04-06 05:58:43,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:43,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:43,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:43,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:43,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:43,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:43,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:43,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65885fbe[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:43,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:43,952 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,953 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,953 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:43,953 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:43,953 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:43,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:43,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:43,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:43,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:43,957 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:43,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:43,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:43,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:43,957 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:43,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:43,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:43,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:43,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:43,959 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:43,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:43,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:43,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:43,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:43,961 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:43,961 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:43,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:43,961 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:43,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:43,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:43,962 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:43,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:43,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:43,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:43,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760783982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760783983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:43,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:43,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760783988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,062 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,064 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,064 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,065 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,066 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,066 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,066 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,066 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,066 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,067 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8abb9d3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,068 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,068 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,069 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,069 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,070 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,072 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,076 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,076 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,076 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760784089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760784089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760784091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,176 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,179 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,180 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,180 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,181 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,182 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@638fbc6[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,185 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,187 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,187 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,187 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,188 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,191 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,191 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,192 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,292 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760784293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,294 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760784293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760784294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,295 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,296 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,296 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,298 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,298 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,298 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,298 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,298 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,300 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec412f9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,301 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,301 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,301 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,302 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,304 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,304 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,308 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,309 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,309 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,385 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406f2d73f2d6e114d4ebec9ae778769a5b4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f2d73f2d6e114d4ebec9ae778769a5b4 2023-04-06 05:58:44,389 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=189, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/f178abf9b7374584ad776b0afe4081c4 2023-04-06 05:58:44,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/d86b4746fd3a418ab83e95f5848193c4 2023-04-06 05:58:44,409 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,411 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,411 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,412 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,412 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,413 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,413 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,413 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,414 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,414 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,414 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@996cd30[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,416 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,416 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,417 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,417 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,417 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,417 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,423 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,424 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,424 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,524 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,526 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,526 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760784525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,527 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,528 DEBUG [Thread-3386] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:44,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,529 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,529 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,529 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c2688f0[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,532 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760784530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,534 DEBUG [Thread-3382] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,537 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,537 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,539 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,540 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44538 deadline: 1680760784596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760784597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760784597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,640 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,642 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,642 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,643 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,643 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,644 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,644 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,645 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,645 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,646 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d81be79[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,647 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,647 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,647 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,648 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,648 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,650 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,650 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,650 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,651 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,651 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,652 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,653 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,654 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,654 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:44,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,754 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,757 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,757 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,758 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,758 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:44,759 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:44,759 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:44,760 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,760 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:44,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13dc24ea[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:44,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:44,761 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,762 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:44,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:44,762 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:44,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,762 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:44,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:44,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,763 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,763 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:44,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:44,764 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:44,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:44,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:44,765 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:44,765 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:44,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,765 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,766 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,767 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,769 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:44,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:44,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:44,769 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,769 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:44,769 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:44,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:44,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/f5bd6d1a8c87429dbb3d3630ae3c2b02 2023-04-06 05:58:44,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/f178abf9b7374584ad776b0afe4081c4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4 2023-04-06 05:58:44,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4, entries=200, sequenceid=189, filesize=32.1 K 2023-04-06 05:58:44,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/d86b4746fd3a418ab83e95f5848193c4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4 2023-04-06 05:58:44,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4, entries=150, sequenceid=189, filesize=11.8 K 2023-04-06 05:58:44,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/f5bd6d1a8c87429dbb3d3630ae3c2b02 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02 2023-04-06 05:58:44,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02, entries=150, sequenceid=189, filesize=11.8 K 2023-04-06 05:58:44,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 0938af50bcbc9322cb599bce38356083 in 848ms, sequenceid=189, compaction requested=true 2023-04-06 05:58:44,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:44,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:44,825 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:44,825 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:44,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:44,825 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:44,826 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,826 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2ff529d2a9b44dddbecdb3991b4c5aa8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=108.0 K 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:44,826 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,826 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/433061bbfec14e02b822921d2901a759, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=47.1 K 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2ff529d2a9b44dddbecdb3991b4c5aa8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720350 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 433061bbfec14e02b822921d2901a759, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720350 2023-04-06 05:58:44,826 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 7bf96660271c4928aaf34210668b4685, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680760720391 2023-04-06 05:58:44,827 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 549668857ace437fa1b478c9d976d564, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680760720391 2023-04-06 05:58:44,827 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ceaf9e7ebf5d417b8edb1348ad01186f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680760722517 2023-04-06 05:58:44,827 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ffeffc78891742f0a5a5870ab6f105a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680760722517 2023-04-06 05:58:44,827 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting f178abf9b7374584ad776b0afe4081c4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:44,827 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d86b4746fd3a418ab83e95f5848193c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:44,833 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:44,835 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#459 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:44,869 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:44,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:44,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:44,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:44,872 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:44,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:44,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:44,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,873 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:44,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:44,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:44,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:44,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:44,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:44,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:44,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:44,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:44,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:44,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:44,874 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:44,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:44,874 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:44,874 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:44,875 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:44,884 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040663d9db0e75014ec3be03d8d0ff723cb2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040663d9db0e75014ec3be03d8d0ff723cb2 2023-04-06 05:58:44,888 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/2710f2a73e2d41b9a62b6d6546451061 2023-04-06 05:58:45,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:45,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760725119, max timestamp=1680760725123Num uniques 859; 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680760725119, max timestamp=1680760725123Num uniques 859; 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680760725119, max timestamp=1680760725123Num uniques 859; 2023-04-06 05:58:45,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-06 05:58:45,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44586 deadline: 1680760785143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760725139, max timestamp=1680760725145Num uniques 1068; 2023-04-06 05:58:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760785144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-06 05:58:45,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760725139, max timestamp=1680760725145Num uniques 1068; 2023-04-06 05:58:45,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760725139, max timestamp=1680760725145Num uniques 1068; 2023-04-06 05:58:45,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-06 05:58:45,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760725181, max timestamp=1680760725193Num uniques 1027; 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760725181, max timestamp=1680760725193Num uniques 1027; 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760725181, max timestamp=1680760725193Num uniques 1027; 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:45,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:45,243 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/f4b9ab6c4ebf4202b55dc996237e2916 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f4b9ab6c4ebf4202b55dc996237e2916 2023-04-06 05:58:45,247 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into f4b9ab6c4ebf4202b55dc996237e2916(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:45,247 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:45,247 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=12, startTime=1680760724825; duration=0sec 2023-04-06 05:58:45,247 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:45,247 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-06 05:58:45,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760725228, max timestamp=1680760725245Num uniques 1036; 2023-04-06 05:58:45,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 05:58:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760725228, max timestamp=1680760725245Num uniques 1036; 2023-04-06 05:58:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 05:58:45,249 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-06 05:58:45,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680760725228, max timestamp=1680760725245Num uniques 1036; 2023-04-06 05:58:45,249 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:45,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-06 05:58:45,249 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,249 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ee633458de3746f78006b1aaa6b2193c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=47.1 K 2023-04-06 05:58:45,250 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ee633458de3746f78006b1aaa6b2193c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680760720350 2023-04-06 05:58:45,250 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 36f575f7abe74dfda466bd28c3a8f4e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1680760720391 2023-04-06 05:58:45,251 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 109feeb580084a1797d88c100deda0cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1680760722517 2023-04-06 05:58:45,251 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f5bd6d1a8c87429dbb3d3630ae3c2b02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:45,262 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:45,295 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/dce1b9c1b17f4ffabd36cdf9263063ff 2023-04-06 05:58:45,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680760725309, max timestamp=1680760725314Num uniques 1045; 2023-04-06 05:58:45,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-06 05:58:45,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760725309, max timestamp=1680760725314Num uniques 1045; 2023-04-06 05:58:45,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:58:45,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680760725309, max timestamp=1680760725314Num uniques 1045; 2023-04-06 05:58:45,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-06 05:58:45,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44560 deadline: 1680760785334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760725328, max timestamp=1680760725336Num uniques 1104; 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760725328, max timestamp=1680760725336Num uniques 1104; 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680760725328, max timestamp=1680760725336Num uniques 1054; 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-06 05:58:45,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760725355, max timestamp=1680760725366Num uniques 1063; 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760725355, max timestamp=1680760725366Num uniques 1063; 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680760725355, max timestamp=1680760725366Num uniques 1063; 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:45,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760725387, max timestamp=1680760725396Num uniques 1072; 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760725387, max timestamp=1680760725396Num uniques 1072; 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680760725387, max timestamp=1680760725396Num uniques 1072; 2023-04-06 05:58:45,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680760725410, max timestamp=1680760725425Num uniques 1081; 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680760725410, max timestamp=1680760725425Num uniques 1081; 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680760725410, max timestamp=1680760725425Num uniques 1081; 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-06 05:58:45,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760725441, max timestamp=1680760725450Num uniques 1040; 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760725441, max timestamp=1680760725450Num uniques 1040; 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680760725441, max timestamp=1680760725450Num uniques 1040; 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 05:58:45,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-04-06 05:58:45,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760785470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680760725465, max timestamp=1680760725472Num uniques 1149; 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680760725465, max timestamp=1680760725472Num uniques 1149; 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680760725465, max timestamp=1680760725472Num uniques 1149; 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-06 05:58:45,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-06 05:58:45,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760785499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760785502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760785574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760785603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760785605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,655 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/211abfcbb6454a3ea53e13e3f402fc4b as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/211abfcbb6454a3ea53e13e3f402fc4b 2023-04-06 05:58:45,658 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 211abfcbb6454a3ea53e13e3f402fc4b(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:45,658 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:45,658 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=12, startTime=1680760724825; duration=0sec 2023-04-06 05:58:45,658 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:45,669 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/7fdae584c1c24a09b289f9c1a4adff57 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7fdae584c1c24a09b289f9c1a4adff57 2023-04-06 05:58:45,672 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 7fdae584c1c24a09b289f9c1a4adff57(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:45,672 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:45,672 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=12, startTime=1680760724825; duration=0sec 2023-04-06 05:58:45,672 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:45,705 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e7f717d987434274814193e202ffc99e 2023-04-06 05:58:45,709 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/2710f2a73e2d41b9a62b6d6546451061 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061 2023-04-06 05:58:45,711 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061, entries=100, sequenceid=197, filesize=18.5 K 2023-04-06 05:58:45,712 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/dce1b9c1b17f4ffabd36cdf9263063ff as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff 2023-04-06 05:58:45,714 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff, entries=100, sequenceid=197, filesize=9.4 K 2023-04-06 05:58:45,715 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e7f717d987434274814193e202ffc99e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e 2023-04-06 05:58:45,717 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e, entries=100, sequenceid=197, filesize=9.4 K 2023-04-06 05:58:45,718 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 0938af50bcbc9322cb599bce38356083 in 844ms, sequenceid=197, compaction requested=false 2023-04-06 05:58:45,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:45,718 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:45,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:45,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:45,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:45,720 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:45,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9c84ec[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:45,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:45,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:45,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,722 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:45,722 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,722 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,723 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:45,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:45,725 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:45,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:45,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:45,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:45,725 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:45,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:45,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:45,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:45,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,729 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,728 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,729 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:45,729 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:45,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:45,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:45,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=154.31 KB heapSize=437.02 KB 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:45,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-06 05:58:45,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760785791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760785806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760785807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,829 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:45,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:45,831 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:45,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:45,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:45,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:45,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:45,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:45,833 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,833 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:45,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:45,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:45,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:45,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:45,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:45,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:45,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4415e887[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:45,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:45,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,836 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:45,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:45,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:45,839 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:45,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:45,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:45,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:45,839 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:45,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,840 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:45,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:45,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:45,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:45,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:45,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:45,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:45,843 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:45,843 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:45,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:45,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760785893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,943 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:45,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:45,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:45,945 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:45,945 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:45,945 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:45,945 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:45,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,946 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:45,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:45,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:45,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:45,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:45,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:45,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:45,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f7f40a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:45,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:45,950 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:45,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:45,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:45,953 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:45,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:45,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:45,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:45,953 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:45,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,955 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:45,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:45,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:45,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:45,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:45,957 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:45,957 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:45,957 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:45,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:45,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,058 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,060 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,060 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,061 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,062 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,063 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,063 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,063 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,063 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:46,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:46,064 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,064 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,064 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,064 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:46,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267e710a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:46,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:46,065 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:46,067 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:46,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:46,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:46,067 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:46,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:46,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:46,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:46,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:46,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:46,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:46,071 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:46,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:46,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,075 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:46,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:46,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,075 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:46,075 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:46,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760786096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760786109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760786112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,175 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,178 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,178 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,179 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,179 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,180 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,180 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,180 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,180 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:46,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:46,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:46,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:46,181 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,181 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,182 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:46,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0d2acd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:46,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:46,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,183 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:46,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:46,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:46,183 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,184 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:46,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:46,186 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:46,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:46,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:46,186 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:46,186 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:46,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,187 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,188 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,189 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,190 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:46,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:46,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,191 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:46,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:46,191 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,191 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,192 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040661fa911d35704fbab50fd118433b164c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661fa911d35704fbab50fd118433b164c 2023-04-06 05:58:46,196 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/1cc01e74774348bb8044ce04eddb2fa1 2023-04-06 05:58:46,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/bb4b0e970e014069a0bddb7594c7bc3c 2023-04-06 05:58:46,291 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,294 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,294 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,295 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,295 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,296 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,296 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,296 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,296 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:46,296 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:46,297 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,297 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:46,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@537b65b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:46,298 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:46,298 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,299 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:46,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:46,299 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:46,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,299 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:46,300 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:46,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,300 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:46,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:46,301 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:46,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:46,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:46,302 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:46,302 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:46,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,302 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,303 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,304 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,305 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,305 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:46,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:46,306 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,306 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,306 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:46,306 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:46,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760786398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,406 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,409 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,409 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,410 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,410 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,411 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,411 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,412 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,412 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,412 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:46,412 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,413 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,413 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8b5553[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:46,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:46,415 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,415 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,415 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:46,416 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:46,416 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:46,416 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,416 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:46,417 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,417 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:46,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:46,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:46,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:46,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:46,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:46,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:46,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,421 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:46,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,424 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,425 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:46,425 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,525 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,528 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,528 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,529 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,529 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,530 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,530 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:46,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:46,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:46,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:46,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,531 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,532 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:46,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba88fa9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:46,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:46,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,533 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:46,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:46,533 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:46,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,533 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:46,534 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,534 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:46,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:46,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:46,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:46,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:46,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:46,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:46,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,536 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:46,537 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:46,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:46,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:46,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,539 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:46,539 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:46,539 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:46,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:46,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/23f713dd0fec4948b42db6d3835d7152 2023-04-06 05:58:46,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760786614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/1cc01e74774348bb8044ce04eddb2fa1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1 2023-04-06 05:58:46,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:46,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760786616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1, entries=150, sequenceid=410, filesize=25.4 K 2023-04-06 05:58:46,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/bb4b0e970e014069a0bddb7594c7bc3c as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c 2023-04-06 05:58:46,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c, entries=150, sequenceid=410, filesize=11.9 K 2023-04-06 05:58:46,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/23f713dd0fec4948b42db6d3835d7152 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152 2023-04-06 05:58:46,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152, entries=150, sequenceid=410, filesize=11.9 K 2023-04-06 05:58:46,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.97 KB/447456, currentSize=26.84 KB/27480 for 0938af50bcbc9322cb599bce38356083 in 847ms, sequenceid=410, compaction requested=true 2023-04-06 05:58:46,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:46,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:46,631 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:46,631 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:46,631 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:46,633 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 71182 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:46,634 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,634 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,634 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/211abfcbb6454a3ea53e13e3f402fc4b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=69.5 K 2023-04-06 05:58:46,634 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f4b9ab6c4ebf4202b55dc996237e2916, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=33.4 K 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 211abfcbb6454a3ea53e13e3f402fc4b, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:46,634 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f4b9ab6c4ebf4202b55dc996237e2916, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:46,635 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 2710f2a73e2d41b9a62b6d6546451061, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680760723980 2023-04-06 05:58:46,635 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting dce1b9c1b17f4ffabd36cdf9263063ff, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680760723980 2023-04-06 05:58:46,635 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 1cc01e74774348bb8044ce04eddb2fa1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:46,635 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bb4b0e970e014069a0bddb7594c7bc3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:46,639 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:46,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:46,641 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:46,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:46,644 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#469 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:46,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:46,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:46,653 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:46,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:46,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:46,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,653 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,654 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:46,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:46,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:46,654 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:46,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:46,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:46,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:46,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:46,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:46,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:46,655 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:46,656 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:46,656 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:46,656 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:46,666 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c6bc66e4af0c47bfad46f2030bf1b7ce to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6bc66e4af0c47bfad46f2030bf1b7ce 2023-04-06 05:58:46,670 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=417, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/5aec9e772bbf473899011f12b604ec36 2023-04-06 05:58:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:46,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760726944, max timestamp=1680760726957Num uniques 859; 2023-04-06 05:58:46,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:46,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760726944, max timestamp=1680760726957Num uniques 859; 2023-04-06 05:58:46,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:46,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760726944, max timestamp=1680760726957Num uniques 859; 2023-04-06 05:58:46,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760726984, max timestamp=1680760726999Num uniques 1018; 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760726984, max timestamp=1680760726999Num uniques 1018; 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680760726984, max timestamp=1680760726999Num uniques 1018; 2023-04-06 05:58:47,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-06 05:58:47,048 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/bcecc67fcfa244708108cf0a594bcb10 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bcecc67fcfa244708108cf0a594bcb10 2023-04-06 05:58:47,054 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into bcecc67fcfa244708108cf0a594bcb10(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:47,054 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,054 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760726631; duration=0sec 2023-04-06 05:58:47,054 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:47,054 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:47,055 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:47,055 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:47,055 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,055 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7fdae584c1c24a09b289f9c1a4adff57, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=33.4 K 2023-04-06 05:58:47,055 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 7fdae584c1c24a09b289f9c1a4adff57, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680760723664 2023-04-06 05:58:47,055 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e7f717d987434274814193e202ffc99e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680760723980 2023-04-06 05:58:47,056 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 23f713dd0fec4948b42db6d3835d7152, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:47,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760727035, max timestamp=1680760727057Num uniques 1027; 2023-04-06 05:58:47,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:47,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760727035, max timestamp=1680760727057Num uniques 1027; 2023-04-06 05:58:47,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680760727035, max timestamp=1680760727057Num uniques 1027; 2023-04-06 05:58:47,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:47,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-06 05:58:47,062 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#472 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:47,078 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/acd8626336d44a72940067897227e5da 2023-04-06 05:58:47,085 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e00bdce19eed4c63ab647c877af062f0 2023-04-06 05:58:47,088 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/5aec9e772bbf473899011f12b604ec36 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36 2023-04-06 05:58:47,091 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36, entries=100, sequenceid=417, filesize=18.6 K 2023-04-06 05:58:47,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/acd8626336d44a72940067897227e5da as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da 2023-04-06 05:58:47,094 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da, entries=100, sequenceid=417, filesize=9.5 K 2023-04-06 05:58:47,095 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e00bdce19eed4c63ab647c877af062f0 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0 2023-04-06 05:58:47,097 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0, entries=100, sequenceid=417, filesize=9.5 K 2023-04-06 05:58:47,098 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=120.76 KB/123660 for 0938af50bcbc9322cb599bce38356083 in 442ms, sequenceid=417, compaction requested=false 2023-04-06 05:58:47,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,098 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:47,098 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=127.47 KB heapSize=363.38 KB 2023-04-06 05:58:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:47,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:47,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:47,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:47,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:47,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:47,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,099 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c748153[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,101 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,102 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,103 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,103 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,106 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,106 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,106 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,109 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,109 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066c0579f1e1be44709c2074a0eb4bb968 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066c0579f1e1be44709c2074a0eb4bb968 2023-04-06 05:58:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,109 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,109 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,110 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,113 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,113 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,113 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,113 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,114 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760787125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,213 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,216 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,217 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,218 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,218 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,219 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,219 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,219 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,220 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c96de8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,220 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,221 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,221 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,221 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,222 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,222 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,222 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,222 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,223 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,224 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,224 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,224 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,224 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,225 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,224 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,225 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,227 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,227 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,227 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,227 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760787230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,327 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,330 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,331 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,332 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,332 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,333 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,333 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,333 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,333 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,334 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,334 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30dc81ef[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,335 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,336 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,336 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,336 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,336 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,337 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,338 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,338 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,339 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,339 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,340 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,341 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,342 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,342 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,342 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,342 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760787432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,442 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,445 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,445 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,445 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,445 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,446 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,446 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,447 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,447 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,447 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,448 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,448 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff6959a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,450 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,450 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,450 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,451 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,451 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,452 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,452 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,452 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,453 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,454 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,456 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,456 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,456 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,456 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,467 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/824016b66ac3482e9cec6bf3df7ef73f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/824016b66ac3482e9cec6bf3df7ef73f 2023-04-06 05:58:47,470 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/c0ded247bd174440abea2101f5fea999 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0ded247bd174440abea2101f5fea999 2023-04-06 05:58:47,472 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 824016b66ac3482e9cec6bf3df7ef73f(size=25.9 K), total size for store is 44.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:47,472 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,472 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760726631; duration=0sec 2023-04-06 05:58:47,472 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:47,474 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into c0ded247bd174440abea2101f5fea999(size=12.3 K), total size for store is 21.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:47,474 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,474 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760726631; duration=0sec 2023-04-06 05:58:47,474 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:47,515 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=488, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/97f2368254cc4fe3871ccfc59afde0ba 2023-04-06 05:58:47,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/90d55dce38504a88ac126e969aa811aa 2023-04-06 05:58:47,556 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,559 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,560 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,560 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,561 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,561 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,561 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,562 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,562 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,562 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,562 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,563 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5710eb2e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,564 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,564 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,564 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,564 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,564 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,564 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,565 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,565 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,567 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,567 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,567 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,567 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,568 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,569 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,571 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,571 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,571 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,571 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,572 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760787617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760787620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,672 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,674 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,678 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,678 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,678 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,678 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,678 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,679 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,679 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18f96a09[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,680 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,681 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,681 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,681 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,682 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,683 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,683 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,683 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,684 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,684 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,685 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,686 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,686 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,687 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,687 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,687 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:47,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760787735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,787 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,790 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,791 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,791 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,791 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,791 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,792 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,792 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,792 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,792 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,792 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,793 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,793 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cfe4300[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,795 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,795 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,795 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,796 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,796 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,798 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,798 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,798 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,799 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,800 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,801 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,801 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,802 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,802 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,802 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,902 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:47,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:47,905 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:47,905 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,905 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:47,905 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:47,905 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,905 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,906 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:47,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:47,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:47,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:47,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:47,907 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:47,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:47,908 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,908 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,908 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,908 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:47,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37df4540[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:47,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:47,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:47,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:47,910 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:47,910 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:47,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:47,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,911 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:47,913 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:47,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:47,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:47,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:47,913 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:47,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:47,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,914 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,915 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:47,917 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:47,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:47,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:47,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:47,917 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:47,918 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:47,918 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:47,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/b1bd5eabc8c14b55a4abed5ddaf7b019 2023-04-06 05:58:47,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/97f2368254cc4fe3871ccfc59afde0ba as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba 2023-04-06 05:58:47,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba, entries=200, sequenceid=488, filesize=32.3 K 2023-04-06 05:58:47,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/90d55dce38504a88ac126e969aa811aa as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa 2023-04-06 05:58:47,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa, entries=150, sequenceid=488, filesize=11.9 K 2023-04-06 05:58:47,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/b1bd5eabc8c14b55a4abed5ddaf7b019 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019 2023-04-06 05:58:47,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019, entries=150, sequenceid=488, filesize=11.9 K 2023-04-06 05:58:47,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~363.33 KB/372048, currentSize=53.67 KB/54960 for 0938af50bcbc9322cb599bce38356083 in 845ms, sequenceid=488, compaction requested=true 2023-04-06 05:58:47,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:47,943 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:47,944 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:47,947 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78574 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:47,947 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:47,947 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:47,947 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:47,947 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,947 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,947 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/824016b66ac3482e9cec6bf3df7ef73f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=76.7 K 2023-04-06 05:58:47,947 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bcecc67fcfa244708108cf0a594bcb10, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=33.7 K 2023-04-06 05:58:47,947 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 824016b66ac3482e9cec6bf3df7ef73f, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:47,948 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bcecc67fcfa244708108cf0a594bcb10, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:47,948 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5aec9e772bbf473899011f12b604ec36, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680760725787 2023-04-06 05:58:47,948 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting acd8626336d44a72940067897227e5da, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680760725787 2023-04-06 05:58:47,948 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 97f2368254cc4fe3871ccfc59afde0ba, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:47,948 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 90d55dce38504a88ac126e969aa811aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:47,953 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:47,955 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#478 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:47,969 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/b2db5e08fafa4c819eef0babfeaf409e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/b2db5e08fafa4c819eef0babfeaf409e 2023-04-06 05:58:47,973 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into b2db5e08fafa4c819eef0babfeaf409e(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:47,973 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:47,973 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760727943; duration=0sec 2023-04-06 05:58:47,973 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:47,973 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:47,974 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:47,974 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:47,974 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:47,974 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0ded247bd174440abea2101f5fea999, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=33.7 K 2023-04-06 05:58:47,974 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c0ded247bd174440abea2101f5fea999, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680760725486 2023-04-06 05:58:47,974 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting e00bdce19eed4c63ab647c877af062f0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680760725787 2023-04-06 05:58:47,975 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting b1bd5eabc8c14b55a4abed5ddaf7b019, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:47,979 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:48,018 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:48,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:48,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:48,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:48,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:48,020 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:48,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:48,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:48,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:48,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:48,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:48,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:48,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:48,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:48,023 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:48,023 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:48,023 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:48,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:48,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760728275, max timestamp=1680760728282Num uniques 859; 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760728275, max timestamp=1680760728282Num uniques 859; 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760728275, max timestamp=1680760728282Num uniques 859; 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:48,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:48,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760788328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,360 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/d935252a9df34c3fa9f932ee8e9bd6dd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d935252a9df34c3fa9f932ee8e9bd6dd 2023-04-06 05:58:48,363 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into d935252a9df34c3fa9f932ee8e9bd6dd(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:48,363 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:48,363 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760727943; duration=0sec 2023-04-06 05:58:48,363 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:48,386 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/1c4735ad4f064a21bec218f1a4f4a744 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1c4735ad4f064a21bec218f1a4f4a744 2023-04-06 05:58:48,389 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 1c4735ad4f064a21bec218f1a4f4a744(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:48,389 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:48,389 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760727944; duration=0sec 2023-04-06 05:58:48,390 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:48,432 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406afa3ef52ff3a47b0a2fd9ebb6f1d47f4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406afa3ef52ff3a47b0a2fd9ebb6f1d47f4 2023-04-06 05:58:48,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760788431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,436 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=502, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/4b278b0f27b3450890fdd7e136447e98 2023-04-06 05:58:48,445 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/307975a9dca5409a8fd8cadf8de42af9 2023-04-06 05:58:48,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44550 deadline: 1680760788556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,558 DEBUG [Thread-3386] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:48,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44584 deadline: 1680760788563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,565 DEBUG [Thread-3382] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:48,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760788633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,852 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/1969c974df6e44599c8294259e4694d1 2023-04-06 05:58:48,855 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/4b278b0f27b3450890fdd7e136447e98 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98 2023-04-06 05:58:48,858 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98, entries=150, sequenceid=502, filesize=25.4 K 2023-04-06 05:58:48,858 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/307975a9dca5409a8fd8cadf8de42af9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9 2023-04-06 05:58:48,861 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9, entries=150, sequenceid=502, filesize=11.9 K 2023-04-06 05:58:48,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/1969c974df6e44599c8294259e4694d1 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1 2023-04-06 05:58:48,864 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1, entries=150, sequenceid=502, filesize=11.9 K 2023-04-06 05:58:48,865 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0938af50bcbc9322cb599bce38356083 in 841ms, sequenceid=502, compaction requested=false 2023-04-06 05:58:48,865 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:48,865 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:48,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:48,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:48,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:48,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:48,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:48,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:48,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:48,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6746810e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:48,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:48,867 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,867 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:48,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:48,868 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,868 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:48,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:48,871 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:48,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:48,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:48,871 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:48,871 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:48,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:48,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:48,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:48,873 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,874 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,875 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:48,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:48,875 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:48,876 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:48,876 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:48,876 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:48,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:48,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:48,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 05:58:48,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:48,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:48,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:48,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:48,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:48,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:48,948 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406770a2180a29c4bd98d74f1e89f1b9b42 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406770a2180a29c4bd98d74f1e89f1b9b42 2023-04-06 05:58:48,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760788957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,976 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:48,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:48,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:48,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:48,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:48,994 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:48,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:48,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:48,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:48,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,994 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,995 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:48,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,995 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:48,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:48,996 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:48,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,003 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,004 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,011 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343c06cd[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,012 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,016 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,016 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,019 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,019 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,020 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,021 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,022 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,022 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,022 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,022 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760789059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,123 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,132 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,133 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae4b509[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,137 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,138 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,139 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,139 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,139 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,140 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:49,143 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,143 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,143 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,143 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,244 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,246 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,246 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,246 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,246 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,247 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,248 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,248 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,248 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,250 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,250 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475c1450[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,252 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,253 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,255 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,255 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,255 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,255 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,255 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,258 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,258 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,258 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,259 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:49,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760789261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,353 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=541, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/c1b1eab666e8410aa852ab27f651d612 2023-04-06 05:58:49,359 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,361 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,362 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,362 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,363 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,363 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,363 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,364 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,364 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,364 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,364 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,365 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,365 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1441b344[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,367 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,368 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,368 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,368 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,369 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,370 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,370 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,372 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,372 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,372 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,372 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,473 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,475 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,475 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,476 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,477 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,477 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,477 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,477 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,477 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,478 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,478 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@401fc56f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,480 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,480 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,481 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,481 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,481 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,481 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,483 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,483 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,483 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,484 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,485 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:49,487 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,488 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,488 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760789566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,588 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,590 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,592 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,595 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ffc6c98[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,598 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,602 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,610 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,610 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:49,611 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,611 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:49,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760789623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,626 DEBUG [Thread-3380] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4123 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:49,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:49,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760789632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,636 DEBUG [Thread-3378] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., hostname=jenkins-hbase20.apache.org,33239,1680760570949, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-06 05:58:49,711 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,713 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,713 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,714 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,714 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,715 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,715 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,715 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,716 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,716 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,716 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,716 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,716 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,717 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48250ba3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,719 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,720 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,720 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,723 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,723 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,724 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:49,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,727 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,728 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/f8041847d8d545beb4f2fb9d27109229 2023-04-06 05:58:49,828 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,830 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,830 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,831 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51674a70[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,835 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,835 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,836 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,836 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,838 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,838 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,839 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,839 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,839 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,842 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,843 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:49,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,843 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,843 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,843 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:49,943 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:49,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:49,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:49,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:49,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:49,946 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:49,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:49,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:49,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,946 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,947 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,947 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:49,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:49,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:49,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:49,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:49,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:49,948 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:49,948 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:49,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,949 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6dd98e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:49,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:49,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:49,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:49,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,951 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:49,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:49,952 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:49,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:49,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:49,954 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:49,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:49,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:49,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:49,954 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:49,955 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:49,956 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:49,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,956 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,957 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:49,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:49,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:49,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:49,958 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:49,958 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:49,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:49,958 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:50,059 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:50,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:50,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:50,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:50,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:50,061 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:50,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:50,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:50,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,061 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,062 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,062 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:50,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:50,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:50,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:50,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:50,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,063 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:50,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:50,067 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:50,068 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,068 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:50,068 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:50,068 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:50,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:50,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:50,068 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:50,069 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:50,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6379f057[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:50,070 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:50,070 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,071 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,071 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:50,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:50,072 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,072 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,072 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,072 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:50,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:50,073 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:50,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:50,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:50,074 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:50,074 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:50,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:50,075 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760790073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:50,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,077 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:50,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:50,078 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:50,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/b497004a2c2742b8ad1ac038d901d693 2023-04-06 05:58:50,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/c1b1eab666e8410aa852ab27f651d612 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612 2023-04-06 05:58:50,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612, entries=150, sequenceid=541, filesize=25.4 K 2023-04-06 05:58:50,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/f8041847d8d545beb4f2fb9d27109229 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229 2023-04-06 05:58:50,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229, entries=150, sequenceid=541, filesize=11.9 K 2023-04-06 05:58:50,178 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:50,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:50,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/b497004a2c2742b8ad1ac038d901d693 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693 2023-04-06 05:58:50,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:50,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:50,182 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:50,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:50,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:50,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,182 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,183 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693, entries=150, sequenceid=541, filesize=11.9 K 2023-04-06 05:58:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:50,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 0938af50bcbc9322cb599bce38356083 in 1245ms, sequenceid=541, compaction requested=true 2023-04-06 05:58:50,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:50,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:50,184 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:50,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:50,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:50,185 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:50,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78684 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:50,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:50,185 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:50,185 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,186 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/b2db5e08fafa4c819eef0babfeaf409e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=76.8 K 2023-04-06 05:58:50,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:50,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:50,185 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:50,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:50,194 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting b2db5e08fafa4c819eef0babfeaf409e, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:50,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:50,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:50,194 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:50,194 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,194 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d935252a9df34c3fa9f932ee8e9bd6dd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.2 K 2023-04-06 05:58:50,194 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:50,195 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 4b278b0f27b3450890fdd7e136447e98, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760727115 2023-04-06 05:58:50,195 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:50,195 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d935252a9df34c3fa9f932ee8e9bd6dd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:50,195 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c1b1eab666e8410aa852ab27f651d612, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:50,195 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 307975a9dca5409a8fd8cadf8de42af9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760727115 2023-04-06 05:58:50,195 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:50,195 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:50,195 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:50,196 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:50,196 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting f8041847d8d545beb4f2fb9d27109229, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:50,223 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:50,236 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#487 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:50,248 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/3cce81f60e05467e91ebff96f196983d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/3cce81f60e05467e91ebff96f196983d 2023-04-06 05:58:50,254 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 3cce81f60e05467e91ebff96f196983d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:50,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,255 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760730184; duration=0sec 2023-04-06 05:58:50,255 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040614233e8fce4e409d905bbe667ae2b44a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040614233e8fce4e409d905bbe667ae2b44a 2023-04-06 05:58:50,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:50,255 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:50,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:50,256 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:50,256 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,256 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1c4735ad4f064a21bec218f1a4f4a744, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.2 K 2023-04-06 05:58:50,257 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1c4735ad4f064a21bec218f1a4f4a744, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1680760727087 2023-04-06 05:58:50,257 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 1969c974df6e44599c8294259e4694d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1680760727115 2023-04-06 05:58:50,257 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting b497004a2c2742b8ad1ac038d901d693, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:50,263 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:50,274 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/a590a59cf13a4212a6de169e2c5f0792 2023-04-06 05:58:50,280 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/9d899e2bec9242a4b420d9f6d8d46454 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/9d899e2bec9242a4b420d9f6d8d46454 2023-04-06 05:58:50,284 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 9d899e2bec9242a4b420d9f6d8d46454(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:50,284 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,284 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760730185; duration=0sec 2023-04-06 05:58:50,285 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/4363e02e665145348af769e9cb66816f 2023-04-06 05:58:50,285 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:50,670 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/c82db9bdee1e4941bdf2cd9bc6872ffc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c82db9bdee1e4941bdf2cd9bc6872ffc 2023-04-06 05:58:50,673 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into c82db9bdee1e4941bdf2cd9bc6872ffc(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:50,673 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,673 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760730184; duration=0sec 2023-04-06 05:58:50,674 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:50,704 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/94d84094059a48b29cbb3d63acacdc4e 2023-04-06 05:58:50,707 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/a590a59cf13a4212a6de169e2c5f0792 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792 2023-04-06 05:58:50,709 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792, entries=100, sequenceid=551, filesize=18.6 K 2023-04-06 05:58:50,710 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/4363e02e665145348af769e9cb66816f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f 2023-04-06 05:58:50,713 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f, entries=100, sequenceid=551, filesize=9.5 K 2023-04-06 05:58:50,713 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/94d84094059a48b29cbb3d63acacdc4e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e 2023-04-06 05:58:50,716 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e, entries=100, sequenceid=551, filesize=9.5 K 2023-04-06 05:58:50,717 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 0938af50bcbc9322cb599bce38356083 in 523ms, sequenceid=551, compaction requested=false 2023-04-06 05:58:50,717 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,717 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:50,717 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:50,718 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,718 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,719 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,719 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:50,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3619788[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:50,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:50,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,720 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:50,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:50,720 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,721 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:50,720 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:50,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:50,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,721 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,721 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:50,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:50,723 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:50,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:50,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:50,723 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:50,723 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:50,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,724 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,725 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,726 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:50,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:50,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:50,727 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,727 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,728 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:50,728 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,828 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:50,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:50,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:50,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:50,831 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:50,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:50,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:50,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,831 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,832 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,832 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:50,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:50,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:50,833 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:50,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:50,833 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,834 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:50,834 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,834 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,835 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:50,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68be695e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:50,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:50,836 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,837 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:50,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,837 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:50,837 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:50,838 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:50,838 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,838 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,838 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:50,840 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:50,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:50,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:50,840 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:50,840 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:50,841 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,841 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,841 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,841 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,842 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,844 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:50,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,845 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:50,845 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:50,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,845 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,945 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:50,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:50,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:50,948 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:50,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:50,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:50,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,948 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,949 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,949 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:50,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:50,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:50,950 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:50,950 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,950 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:50,950 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:50,951 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:50,951 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,951 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,952 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:50,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699e7c6e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:50,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:50,953 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:50,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:50,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:50,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,954 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:50,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:50,954 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,955 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:50,954 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:50,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:50,958 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:50,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:50,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:50,958 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:50,958 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:50,959 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,959 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,959 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,960 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:50,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,961 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:50,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:50,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:50,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:50,963 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:50,963 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:50,963 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:50,963 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:50,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:50,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,063 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,066 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,066 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,067 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,067 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,068 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,068 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,069 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,069 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:51,070 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,070 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,071 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,071 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c9c242[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,072 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,073 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,073 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,073 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,073 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,073 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,074 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,076 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,076 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,076 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,077 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,078 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,080 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,080 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,080 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,080 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:51,121 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-06 05:58:51,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:51,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:51,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:51,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:51,122 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:51,122 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:51,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760731160, max timestamp=1680760731177Num uniques 859; 2023-04-06 05:58:51,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760731160, max timestamp=1680760731177Num uniques 859; 2023-04-06 05:58:51,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760731160, max timestamp=1680760731177Num uniques 859; 2023-04-06 05:58:51,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:51,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:51,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:51,180 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,183 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,183 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,184 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,185 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,185 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,186 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,190 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,190 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,191 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,191 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1bd0a4[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,193 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,193 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,193 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,194 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,194 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,194 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,197 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,197 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,197 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,197 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,198 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,198 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,199 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,202 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,202 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,202 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,202 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:51,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760791224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,303 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,306 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,306 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,307 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,308 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,308 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f4a420[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,310 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,311 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,311 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,311 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,312 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,312 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,312 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,312 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,314 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,314 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,315 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,315 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,317 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,317 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,318 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,318 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,318 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,318 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:51,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760791326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,418 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,421 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,422 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,422 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,423 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,423 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,423 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,423 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,424 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,424 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,424 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,424 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b13cf7c[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,427 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,429 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,429 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,429 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,429 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,429 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,430 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,430 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,430 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,431 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,432 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,432 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,432 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,433 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,433 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:51,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760791529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,531 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304069aa1c4d7c1f642f7a50bcd76961b11ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069aa1c4d7c1f642f7a50bcd76961b11ee 2023-04-06 05:58:51,533 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,534 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=567, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/d139149a958f426aa1e12429f63aee9a 2023-04-06 05:58:51,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,536 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,536 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,538 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,538 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,539 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,540 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,540 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,540 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,540 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,540 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,541 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,541 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,541 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,542 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4043cf8e[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,543 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,543 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,544 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,544 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,544 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,544 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,544 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,545 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,547 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,547 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,550 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,552 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,553 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,553 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,653 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,655 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,655 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,655 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,655 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,656 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,656 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,657 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,657 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,658 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,658 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,659 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,659 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,660 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c9b9d2[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,661 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,661 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,661 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,661 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,661 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,661 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,662 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,662 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,664 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,664 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,665 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,666 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,667 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,667 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,668 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,668 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,768 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,770 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,770 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,771 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,771 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,772 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,772 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,773 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,773 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,773 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,773 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,774 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,774 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2068d2cf[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,775 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,775 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,775 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,776 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,776 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,776 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,778 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,779 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,779 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,781 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,782 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,782 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,783 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,783 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:51,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760791832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,883 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:51,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:51,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:51,887 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:51,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:51,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:51,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,887 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,888 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,888 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:51,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:51,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:51,889 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,889 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:51,889 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:51,889 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:51,889 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:51,890 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,890 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:51,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d58191[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:51,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:51,891 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,892 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:51,892 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:51,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,892 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,893 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,893 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:51,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:51,895 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:51,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:51,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:51,895 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:51,895 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:51,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:51,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,896 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,897 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:51,898 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:51,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:51,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,899 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,899 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:51,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:51,899 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:51,899 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:51,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/8b1d35ab22944c80bbc7ef8a51afebd5 2023-04-06 05:58:51,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/d4ed9970fdf948919a7f06d4b0c69e27 2023-04-06 05:58:51,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/d139149a958f426aa1e12429f63aee9a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a 2023-04-06 05:58:51,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a, entries=350, sequenceid=567, filesize=52.9 K 2023-04-06 05:58:51,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/8b1d35ab22944c80bbc7ef8a51afebd5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5 2023-04-06 05:58:51,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5, entries=100, sequenceid=567, filesize=9.5 K 2023-04-06 05:58:51,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/d4ed9970fdf948919a7f06d4b0c69e27 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27 2023-04-06 05:58:51,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27, entries=100, sequenceid=567, filesize=9.5 K 2023-04-06 05:58:51,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0938af50bcbc9322cb599bce38356083 in 845ms, sequenceid=567, compaction requested=true 2023-04-06 05:58:51,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:51,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:51,966 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:51,966 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:51,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:51,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:51,967 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99902 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:51,967 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:51,967 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,967 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32318 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:51,967 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c82db9bdee1e4941bdf2cd9bc6872ffc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=97.6 K 2023-04-06 05:58:51,967 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:51,967 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:51,967 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/3cce81f60e05467e91ebff96f196983d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=31.6 K 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting c82db9bdee1e4941bdf2cd9bc6872ffc, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 3cce81f60e05467e91ebff96f196983d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting a590a59cf13a4212a6de169e2c5f0792, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680760728954 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 4363e02e665145348af769e9cb66816f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680760728954 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting d139149a958f426aa1e12429f63aee9a, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760731089 2023-04-06 05:58:51,968 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 8b1d35ab22944c80bbc7ef8a51afebd5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760731117 2023-04-06 05:58:51,973 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:51,980 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#496 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:51,999 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:52,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:52,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:52,002 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:52,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:52,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:52,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:52,002 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,003 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:52,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:52,003 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,004 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:52,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:52,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:52,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:52,004 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:52,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:52,005 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:52,005 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:52,005 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:52,006 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:52,006 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:52,021 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304066356407a9d8447e1a39446f8779d08f1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066356407a9d8447e1a39446f8779d08f1 2023-04-06 05:58:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:52,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:52,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:52,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760792352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,383 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/9c4c5ce941284f61bc765f0e5a59d976 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/9c4c5ce941284f61bc765f0e5a59d976 2023-04-06 05:58:52,386 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 9c4c5ce941284f61bc765f0e5a59d976(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:52,386 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:52,386 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760731966; duration=0sec 2023-04-06 05:58:52,386 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:52,386 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:52,387 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 32318 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:52,387 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:52,387 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:52,387 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/9d899e2bec9242a4b420d9f6d8d46454, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=31.6 K 2023-04-06 05:58:52,387 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9d899e2bec9242a4b420d9f6d8d46454, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1680760728311 2023-04-06 05:58:52,388 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 94d84094059a48b29cbb3d63acacdc4e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680760728954 2023-04-06 05:58:52,388 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting d4ed9970fdf948919a7f06d4b0c69e27, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760731117 2023-04-06 05:58:52,403 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:52,423 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/411c5d5c2f6e4c67a037c4de020e0943 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/411c5d5c2f6e4c67a037c4de020e0943 2023-04-06 05:58:52,428 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=604, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/9ddd4ed5084c4efa8c56da025f43a679 2023-04-06 05:58:52,428 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 411c5d5c2f6e4c67a037c4de020e0943(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:52,428 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:52,428 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760731966; duration=0sec 2023-04-06 05:58:52,428 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:52,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:52,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760792455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:52,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760792659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,834 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/ab04ceb1865a49848e801fee448f9831 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ab04ceb1865a49848e801fee448f9831 2023-04-06 05:58:52,837 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/5ab550e8dba44b76bb3c3ef40130a012 2023-04-06 05:58:52,837 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into ab04ceb1865a49848e801fee448f9831(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:52,837 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:52,838 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760731966; duration=0sec 2023-04-06 05:58:52,838 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:52,846 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e6d429dd440f4137ae7217566540a054 2023-04-06 05:58:52,851 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/9ddd4ed5084c4efa8c56da025f43a679 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679 2023-04-06 05:58:52,853 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679, entries=150, sequenceid=604, filesize=25.4 K 2023-04-06 05:58:52,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/5ab550e8dba44b76bb3c3ef40130a012 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012 2023-04-06 05:58:52,857 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:58:52,857 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/e6d429dd440f4137ae7217566540a054 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054 2023-04-06 05:58:52,860 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054, entries=150, sequenceid=604, filesize=11.9 K 2023-04-06 05:58:52,861 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 0938af50bcbc9322cb599bce38356083 in 856ms, sequenceid=604, compaction requested=false 2023-04-06 05:58:52,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:52,861 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:52,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:52,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:52,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:52,861 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:52,862 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,862 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:52,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b340f1a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:52,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:52,863 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,865 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,865 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:52,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:52,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:52,865 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:52,866 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:52,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,866 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:52,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:52,868 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:52,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:52,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:52,868 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:52,868 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:52,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,869 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:52,869 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,870 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,872 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:52,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:52,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:52,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:52,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,872 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,872 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:52,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:52,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,872 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:52,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:52,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:52,972 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:52,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:52,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:52,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:52,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:52,975 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:52,975 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:52,975 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:52,975 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:52,975 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,975 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,976 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:52,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:52,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,976 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:52,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:52,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:52,977 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:52,977 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:52,977 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:52,977 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:52,977 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:52,978 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,978 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32aa3487[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:52,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:52,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,980 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,980 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:52,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,980 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:52,980 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:52,981 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:52,981 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,981 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,981 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:52,981 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:52,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:52,983 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:52,983 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:52,983 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:52,983 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:52,983 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:52,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:52,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,984 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,985 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,986 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:52,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:52,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:52,987 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:52,987 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:52,987 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:52,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:52,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:52,987 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:52,988 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:52,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760732983, max timestamp=1680760733002Num uniques 859; 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760732983, max timestamp=1680760733002Num uniques 859; 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680760732983, max timestamp=1680760733002Num uniques 859; 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:53,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-06 05:58:53,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,087 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:53,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:53,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:53,090 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,090 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,091 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,091 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:53,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:53,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:53,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:53,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:53,092 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:53,093 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,094 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:53,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b266b7[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:53,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:53,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,095 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:53,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:53,095 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:53,095 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,096 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:53,096 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,096 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:53,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:53,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:53,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:53,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:53,098 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:53,098 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:53,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,099 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,099 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,100 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,102 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:53,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:53,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:53,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,102 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,103 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,103 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:53,103 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:53,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,203 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:53,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:53,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:53,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:53,205 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:53,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:53,205 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:53,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,206 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,206 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,207 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:53,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:53,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:53,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:53,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:53,208 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:53,209 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,209 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:53,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db2acb9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:53,210 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:53,210 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,211 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:53,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,211 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:53,211 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:53,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,212 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:53,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:53,213 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:53,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:53,213 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:53,214 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:53,214 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,215 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,215 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,216 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,217 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,218 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,218 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:53,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:53,218 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,218 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,318 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:53,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:53,321 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,321 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,322 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,322 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:53,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:53,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,323 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:53,324 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,324 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:53,324 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:53,324 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:53,324 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:53,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,325 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,325 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:53,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f53eca[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:53,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:53,326 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,326 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:53,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,327 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:53,327 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:53,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:53,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:53,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:53,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:53,329 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:53,329 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:53,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,330 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,330 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,331 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,332 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,333 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:53,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:53,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,333 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,333 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:53,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,333 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:53,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:53,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,372 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a01f7e7656c34d9786d4a3dfc32d699e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a01f7e7656c34d9786d4a3dfc32d699e 2023-04-06 05:58:53,376 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=618, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/978356407e5b46f29d7e1877cefeb21d 2023-04-06 05:58:53,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/ca29240834c4413e86f095cd31e63aa2 2023-04-06 05:58:53,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/bad6ef92c1ef4677bc29bbd2a815cff9 2023-04-06 05:58:53,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/978356407e5b46f29d7e1877cefeb21d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d 2023-04-06 05:58:53,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d, entries=200, sequenceid=618, filesize=32.3 K 2023-04-06 05:58:53,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/ca29240834c4413e86f095cd31e63aa2 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2 2023-04-06 05:58:53,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2, entries=150, sequenceid=618, filesize=11.9 K 2023-04-06 05:58:53,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/bad6ef92c1ef4677bc29bbd2a815cff9 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9 2023-04-06 05:58:53,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9, entries=150, sequenceid=618, filesize=11.9 K 2023-04-06 05:58:53,405 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0938af50bcbc9322cb599bce38356083 in 443ms, sequenceid=618, compaction requested=true 2023-04-06 05:58:53,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:53,405 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:53,405 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:53,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:53,405 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:53,405 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:53,405 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:53,405 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,406 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:53,406 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/411c5d5c2f6e4c67a037c4de020e0943, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=83.9 K 2023-04-06 05:58:53,406 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,406 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/9c4c5ce941284f61bc765f0e5a59d976, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.4 K 2023-04-06 05:58:53,406 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 411c5d5c2f6e4c67a037c4de020e0943, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760728956 2023-04-06 05:58:53,406 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 9c4c5ce941284f61bc765f0e5a59d976, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760728956 2023-04-06 05:58:53,406 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 9ddd4ed5084c4efa8c56da025f43a679, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760731213 2023-04-06 05:58:53,415 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 978356407e5b46f29d7e1877cefeb21d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:53,415 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5ab550e8dba44b76bb3c3ef40130a012, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760731213 2023-04-06 05:58:53,416 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ca29240834c4413e86f095cd31e63aa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:53,420 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:53,424 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#504 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:53,428 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/29134fdc14fe421b86c74b547a791d5f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29134fdc14fe421b86c74b547a791d5f 2023-04-06 05:58:53,433 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:53,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:53,439 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 29134fdc14fe421b86c74b547a791d5f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:53,439 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:53,439 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760733405; duration=0sec 2023-04-06 05:58:53,439 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:53,439 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:53,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:53,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:53,439 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:53,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:53,439 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:53,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,440 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:53,440 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:53,440 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,440 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ab04ceb1865a49848e801fee448f9831, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.4 K 2023-04-06 05:58:53,440 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,440 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting ab04ceb1865a49848e801fee448f9831, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1680760728956 2023-04-06 05:58:53,440 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,441 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e6d429dd440f4137ae7217566540a054, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680760731213 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,441 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting bad6ef92c1ef4677bc29bbd2a815cff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:53,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:53,442 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:53,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:53,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,442 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:53,442 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-06 05:58:53,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:53,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:53,442 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:53,443 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:53,443 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:53,443 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-06 05:58:53,447 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:53,463 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040611730eb17d1047b29d2551919e9910f7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040611730eb17d1047b29d2551919e9910f7 2023-04-06 05:58:53,467 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=655, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5 2023-04-06 05:58:53,479 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/adc7d42e475947428a6f49f1b1da1a66 2023-04-06 05:58:53,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:53,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:53,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760793660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760793660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760793765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760793765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,844 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/ca24592675774c9d9f3a99bf5bd3b542 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ca24592675774c9d9f3a99bf5bd3b542 2023-04-06 05:58:53,847 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into ca24592675774c9d9f3a99bf5bd3b542(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:53,847 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:53,847 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760733405; duration=0sec 2023-04-06 05:58:53,847 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:53,860 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/c0d55711a46a4ce58a7d86408b317430 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0d55711a46a4ce58a7d86408b317430 2023-04-06 05:58:53,863 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into c0d55711a46a4ce58a7d86408b317430(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:53,863 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:53,863 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760733405; duration=0sec 2023-04-06 05:58:53,864 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:53,887 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/70bdf78c631247f39a0dc9ff25a1bf9e 2023-04-06 05:58:53,890 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5 2023-04-06 05:58:53,893 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5, entries=150, sequenceid=655, filesize=25.4 K 2023-04-06 05:58:53,893 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/adc7d42e475947428a6f49f1b1da1a66 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66 2023-04-06 05:58:53,896 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66, entries=150, sequenceid=655, filesize=11.9 K 2023-04-06 05:58:53,897 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/70bdf78c631247f39a0dc9ff25a1bf9e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e 2023-04-06 05:58:53,899 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e, entries=150, sequenceid=655, filesize=11.9 K 2023-04-06 05:58:53,900 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 0938af50bcbc9322cb599bce38356083 in 458ms, sequenceid=655, compaction requested=false 2023-04-06 05:58:53,900 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:53,900 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:53,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:53,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:53,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:53,900 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:53,901 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,901 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,902 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:53,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65983c9f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:53,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:53,903 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,903 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:53,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:53,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:53,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,904 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:53,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:53,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,904 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:53,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:53,906 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:53,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:53,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:53,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:53,906 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:53,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:53,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:53,908 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:53,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:53,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:53,911 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:53,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:53,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:53,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:53,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:53,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:53,911 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:53,911 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:53,970 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:53,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:53,980 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406c65ddb026255420b849fe82b95dc8ee7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c65ddb026255420b849fe82b95dc8ee7 2023-04-06 05:58:53,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760793988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760793989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:53,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:53,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760793990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,011 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,014 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,014 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,015 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,015 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,015 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,015 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,016 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,016 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,017 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,017 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22586945[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,019 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,019 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,019 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,020 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,020 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,021 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,021 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,023 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,023 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,023 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,023 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,023 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,024 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,024 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,025 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,025 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,027 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,029 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,029 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,029 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,029 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760794092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760794092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760794093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,129 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,132 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,132 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,132 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,133 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,134 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,134 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,135 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,135 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a17ded3[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,136 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,137 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,137 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,137 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,138 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,138 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,140 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,140 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,140 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,141 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,141 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,142 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,144 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,144 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,144 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,144 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,145 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,244 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,247 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,247 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,248 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,248 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,249 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,249 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,250 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,250 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,250 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,250 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15bba435[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,251 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,251 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,251 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,251 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,251 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,252 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,252 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,254 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,254 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,254 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,254 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,254 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,254 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,254 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,254 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,255 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,256 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,257 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,257 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,257 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,257 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760794295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760794296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760794296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,357 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,360 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,360 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,361 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,361 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,362 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,362 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,362 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,362 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,363 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,363 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,364 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,364 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31631ed9[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,365 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,366 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,366 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,366 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,367 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,367 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,369 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,369 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,369 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,370 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,371 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,373 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,373 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,374 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,374 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,374 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,389 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=669, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/dea6e10912844ff2a60c1458c051b02a 2023-04-06 05:58:54,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/597bd7260556474f8cc39ba7d7faae50 2023-04-06 05:58:54,474 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,476 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,476 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,477 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,477 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,478 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,479 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,479 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,479 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,479 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,479 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,480 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,480 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50905bf1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,481 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,482 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,482 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,482 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,484 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,484 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,484 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,485 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,486 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,487 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,488 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,488 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,488 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,488 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,589 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,591 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,591 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,592 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,592 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,593 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,593 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,594 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,595 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,595 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5475f676[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,596 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,597 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,597 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,597 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,598 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,598 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760794598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,600 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,600 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,601 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,601 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760794599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760794599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,603 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,605 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,605 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,605 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,605 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,705 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,708 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,708 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,709 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,709 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,710 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,710 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,726 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,726 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,726 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:54,726 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:54,728 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,728 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:54,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c276ac1[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:54,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:54,729 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,730 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:54,730 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:54,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,730 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:54,731 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,731 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:54,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:54,733 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:54,733 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:54,733 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:54,734 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:54,735 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,736 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:54,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:54,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,737 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,737 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:54,738 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:54,738 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,738 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:54,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/38522ca9c7d84512b91f390fba6729ee 2023-04-06 05:58:54,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/dea6e10912844ff2a60c1458c051b02a as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a 2023-04-06 05:58:54,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a, entries=200, sequenceid=669, filesize=32.3 K 2023-04-06 05:58:54,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/597bd7260556474f8cc39ba7d7faae50 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50 2023-04-06 05:58:54,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50, entries=150, sequenceid=669, filesize=11.9 K 2023-04-06 05:58:54,815 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/38522ca9c7d84512b91f390fba6729ee as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee 2023-04-06 05:58:54,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee, entries=150, sequenceid=669, filesize=11.9 K 2023-04-06 05:58:54,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0938af50bcbc9322cb599bce38356083 in 849ms, sequenceid=669, compaction requested=true 2023-04-06 05:58:54,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:54,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:54,819 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:54,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:54,819 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:54,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:54,820 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:54,820 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ca24592675774c9d9f3a99bf5bd3b542, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=84.0 K 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:54,820 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,820 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29134fdc14fe421b86c74b547a791d5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.5 K 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting ca24592675774c9d9f3a99bf5bd3b542, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:54,820 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting e3fd269d7d9b4c3eb6cbb5a7d16577a5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1680760733021 2023-04-06 05:58:54,821 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 29134fdc14fe421b86c74b547a791d5f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:54,821 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting dea6e10912844ff2a60c1458c051b02a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733655 2023-04-06 05:58:54,821 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting adc7d42e475947428a6f49f1b1da1a66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1680760733021 2023-04-06 05:58:54,821 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 597bd7260556474f8cc39ba7d7faae50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733657 2023-04-06 05:58:54,826 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#514 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:54,828 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#513 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:54,834 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/5de19d6830da401e8f6a8d4719adcb5f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5de19d6830da401e8f6a8d4719adcb5f 2023-04-06 05:58:54,838 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:54,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:54,839 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 5de19d6830da401e8f6a8d4719adcb5f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:54,839 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:54,839 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760734819; duration=0sec 2023-04-06 05:58:54,839 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:54,839 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:54,840 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:54,840 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:54,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:54,840 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,840 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0d55711a46a4ce58a7d86408b317430, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.5 K 2023-04-06 05:58:54,840 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting c0d55711a46a4ce58a7d86408b317430, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680760732342 2023-04-06 05:58:54,841 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 70bdf78c631247f39a0dc9ff25a1bf9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1680760733021 2023-04-06 05:58:54,841 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 38522ca9c7d84512b91f390fba6729ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733657 2023-04-06 05:58:54,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:54,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:54,844 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:54,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:54,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:54,844 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,845 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,845 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:54,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,846 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:54,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:54,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:54,846 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:54,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:54,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:54,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:54,849 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/aac1f8a4c1b2420597c1fe170a8aef76 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/aac1f8a4c1b2420597c1fe170a8aef76 2023-04-06 05:58:54,852 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into aac1f8a4c1b2420597c1fe170a8aef76(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:54,852 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:54,852 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760734819; duration=0sec 2023-04-06 05:58:54,852 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:54,854 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:54,854 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:54,854 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:54,854 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:54,865 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406e66f5944525e453188dad0d22f21866b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e66f5944525e453188dad0d22f21866b 2023-04-06 05:58:54,871 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=691, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/5a25de6a65484c4cb2c60d7c4b3ac3de 2023-04-06 05:58:54,879 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/e764eff398da4f35a2acca44ec7cc500 2023-04-06 05:58:55,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:55,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760795112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760795113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760795113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760795215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760795215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760795215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,258 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/94324eddddf446cd89d1c65a023884f6 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94324eddddf446cd89d1c65a023884f6 2023-04-06 05:58:55,261 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 94324eddddf446cd89d1c65a023884f6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:55,261 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:55,261 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760734819; duration=0sec 2023-04-06 05:58:55,261 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:55,287 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/97f00f0d633c47de91d8d63dfb3ea224 2023-04-06 05:58:55,291 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/5a25de6a65484c4cb2c60d7c4b3ac3de as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de 2023-04-06 05:58:55,295 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de, entries=150, sequenceid=691, filesize=25.4 K 2023-04-06 05:58:55,296 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/e764eff398da4f35a2acca44ec7cc500 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500 2023-04-06 05:58:55,299 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500, entries=150, sequenceid=691, filesize=11.9 K 2023-04-06 05:58:55,300 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/97f00f0d633c47de91d8d63dfb3ea224 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224 2023-04-06 05:58:55,303 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224, entries=150, sequenceid=691, filesize=11.9 K 2023-04-06 05:58:55,304 INFO [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 0938af50bcbc9322cb599bce38356083 in 450ms, sequenceid=691, compaction requested=false 2023-04-06 05:58:55,305 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:55,305 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,305 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,306 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,306 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,306 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,307 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@351b3a4a[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,309 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,309 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,309 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,310 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,310 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,310 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,310 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,312 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,312 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,312 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,312 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,312 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,313 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,313 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,314 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,316 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:55,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,316 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,316 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,316 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,417 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:55,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:55,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-06 05:58:55,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:55,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:55,419 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:55,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:55,419 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:55,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:55,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:55,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,420 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,420 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,421 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:55,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:55,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:55,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:55,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,422 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,422 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,423 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,423 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,424 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,424 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6786a0cb[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,425 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,426 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,426 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,427 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,427 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,429 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304062ba214d963fd452f867ace94455a20a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062ba214d963fd452f867ace94455a20a1 2023-04-06 05:58:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,429 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,430 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,430 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,430 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,430 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,441 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,441 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,443 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760795441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760795442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,444 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,445 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,445 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,445 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,445 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:55,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760795443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,545 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:55,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:55,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760795544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760795545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760795546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:55,547 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:55,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:55,547 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:55,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,548 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,548 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,549 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:55,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:55,550 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,550 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:55,550 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,550 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,550 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,551 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,551 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbb3c0b[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,552 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,553 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,553 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,553 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,554 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,554 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,555 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,555 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,556 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,556 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,556 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,556 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,557 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,557 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,558 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,559 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:55,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,559 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,559 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,560 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,660 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:55,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:55,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:55,662 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:55,662 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,662 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:55,662 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:55,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,663 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,663 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,664 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:55,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:55,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:55,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:55,665 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,665 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:55,665 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,665 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,665 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,666 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,666 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6d2c3f[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,667 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,667 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,669 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,669 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,669 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,670 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,670 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,671 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,671 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,671 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,672 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,672 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,672 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,672 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,672 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,674 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,675 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,676 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,676 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,677 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,677 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,677 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,677 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33239] ipc.CallRunner(144): callId: 483 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760795747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760795747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:55,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760795748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,777 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:55,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:55,779 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:55,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:55,779 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:55,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,780 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,780 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,781 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:55,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:55,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:55,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:55,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,781 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:55,782 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,782 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:55,782 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,782 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,782 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,783 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,783 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,783 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,783 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57489872[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,784 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,784 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,784 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,785 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,785 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,785 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,785 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,786 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,787 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,788 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,788 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,788 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,788 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,788 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,789 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,790 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,791 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,792 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,792 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,798 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,798 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,843 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=708, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/47799ca24a5c4d73b085c7974f432908 2023-04-06 05:58:55,898 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:55,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-06 05:58:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-06 05:58:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-06 05:58:55,901 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-06 05:58:55,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-06 05:58:55,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-06 05:58:55,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,901 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,902 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,902 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-06 05:58:55,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-06 05:58:55,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-06 05:58:55,903 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,903 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. started... 2023-04-06 05:58:55,903 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. as already flushing 2023-04-06 05:58:55,903 DEBUG [rs(jenkins-hbase20.apache.org,33239,1680760570949)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-06 05:58:55,903 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-06 05:58:55,904 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,904 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,905 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-06 05:58:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,33239,1680760570949' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-06 05:58:55,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51aeaa8[Count = 0] remaining members to acquire global barrier 2023-04-06 05:58:55,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-06 05:58:55,906 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,906 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-06 05:58:55,907 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,907 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,33239,1680760570949' in zk 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-06 05:58:55,907 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,907 DEBUG [member: 'jenkins-hbase20.apache.org,33239,1680760570949' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-06 05:58:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,33239,1680760570949': 2023-04-06 05:58:55,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,33239,1680760570949' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-06 05:58:55,909 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-06 05:58:55,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-06 05:58:55,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-06 05:58:55,909 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-06 05:58:55,909 INFO [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-06 05:58:55,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,910 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-06 05:58:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-06 05:58:55,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-06 05:58:55,911 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-06 05:58:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,912 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-06 05:58:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-06 05:58:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-06 05:58:55,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-06 05:58:55,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-06 05:58:55,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-06 05:58:55,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-06 05:58:55,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,913 DEBUG [(jenkins-hbase20.apache.org,35907,1680760569837)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-06 05:58:55,913 DEBUG [Thread-3388] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-06 05:58:55,913 DEBUG [Thread-3388] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,913 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-06 05:58:55,914 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-06 05:58:55,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-06 05:58:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-06 05:58:56,006 DEBUG [Thread-3389] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x070ab5f5 to 127.0.0.1:55165 2023-04-06 05:58:56,006 DEBUG [Thread-3393] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28ac2b09 to 127.0.0.1:55165 2023-04-06 05:58:56,006 DEBUG [Thread-3389] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:56,006 DEBUG [Thread-3393] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:56,009 DEBUG [Thread-3397] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43591f8a to 127.0.0.1:55165 2023-04-06 05:58:56,009 DEBUG [Thread-3397] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:56,009 DEBUG [Thread-3395] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bf1c01c to 127.0.0.1:55165 2023-04-06 05:58:56,010 DEBUG [Thread-3395] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:56,010 DEBUG [Thread-3391] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2daeb795 to 127.0.0.1:55165 2023-04-06 05:58:56,010 DEBUG [Thread-3391] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:56,013 DEBUG [Thread-3388] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-06 05:58:56,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-06 05:58:56,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760796050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760796050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760796052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/aba5990e18a8468197a8b9776a2b6f60 2023-04-06 05:58:56,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44560 deadline: 1680760796553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-06 05:58:56,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33239] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44538 deadline: 1680760796554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44586 deadline: 1680760796554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:56,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/a896c173f4e34787ba70a27f84375684 2023-04-06 05:58:56,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/47799ca24a5c4d73b085c7974f432908 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908 2023-04-06 05:58:56,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908, entries=150, sequenceid=708, filesize=25.4 K 2023-04-06 05:58:56,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/aba5990e18a8468197a8b9776a2b6f60 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60 2023-04-06 05:58:56,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60, entries=150, sequenceid=708, filesize=11.9 K 2023-04-06 05:58:56,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/a896c173f4e34787ba70a27f84375684 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684 2023-04-06 05:58:56,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684, entries=150, sequenceid=708, filesize=11.9 K 2023-04-06 05:58:56,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0938af50bcbc9322cb599bce38356083 in 1267ms, sequenceid=708, compaction requested=true 2023-04-06 05:58:56,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:56,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:56,686 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:56,686 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:56,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:56,686 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:56,687 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79092 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:56,687 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/A is initiating minor compaction (all files) 2023-04-06 05:58:56,687 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:56,687 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/A in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:56,687 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/B is initiating minor compaction (all files) 2023-04-06 05:58:56,687 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/aac1f8a4c1b2420597c1fe170a8aef76, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=77.2 K 2023-04-06 05:58:56,687 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/B in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:56,688 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5de19d6830da401e8f6a8d4719adcb5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.6 K 2023-04-06 05:58:56,688 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting aac1f8a4c1b2420597c1fe170a8aef76, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733657 2023-04-06 05:58:56,688 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 5de19d6830da401e8f6a8d4719adcb5f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733657 2023-04-06 05:58:56,688 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 5a25de6a65484c4cb2c60d7c4b3ac3de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680760733980 2023-04-06 05:58:56,689 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting e764eff398da4f35a2acca44ec7cc500, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680760733980 2023-04-06 05:58:56,689 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] compactions.Compactor(207): Compacting 47799ca24a5c4d73b085c7974f432908, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1680760735109 2023-04-06 05:58:56,689 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting aba5990e18a8468197a8b9776a2b6f60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1680760735109 2023-04-06 05:58:56,697 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#B#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:56,700 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#A#compaction#522 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:57,109 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/78d44b2b01f24a4cba77f335cf91b16d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/78d44b2b01f24a4cba77f335cf91b16d 2023-04-06 05:58:57,115 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/B of 0938af50bcbc9322cb599bce38356083 into 78d44b2b01f24a4cba77f335cf91b16d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:57,115 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:57,115 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/B, priority=13, startTime=1680760736686; duration=0sec 2023-04-06 05:58:57,115 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-06 05:58:57,115 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-06 05:58:57,117 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-06 05:58:57,117 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1912): 0938af50bcbc9322cb599bce38356083/C is initiating minor compaction (all files) 2023-04-06 05:58:57,117 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0938af50bcbc9322cb599bce38356083/C in TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:57,117 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94324eddddf446cd89d1c65a023884f6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684] into tmpdir=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp, totalSize=36.6 K 2023-04-06 05:58:57,118 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 94324eddddf446cd89d1c65a023884f6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1680760733657 2023-04-06 05:58:57,118 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting 97f00f0d633c47de91d8d63dfb3ea224, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680760733980 2023-04-06 05:58:57,118 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] compactions.Compactor(207): Compacting a896c173f4e34787ba70a27f84375684, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1680760735109 2023-04-06 05:58:57,120 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/646393a34eab40539e75c4dd23ba629f as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/646393a34eab40539e75c4dd23ba629f 2023-04-06 05:58:57,125 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/A of 0938af50bcbc9322cb599bce38356083 into 646393a34eab40539e75c4dd23ba629f(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:57,125 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:57,125 INFO [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/A, priority=13, startTime=1680760736686; duration=0sec 2023-04-06 05:58:57,125 DEBUG [RS:0;jenkins-hbase20:33239-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:57,125 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] throttle.PressureAwareThroughputController(145): 0938af50bcbc9322cb599bce38356083#C#compaction#524 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-06 05:58:57,133 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/4aaf41eeddc34f339adb73b7a8819ddc as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/4aaf41eeddc34f339adb73b7a8819ddc 2023-04-06 05:58:57,138 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0938af50bcbc9322cb599bce38356083/C of 0938af50bcbc9322cb599bce38356083 into 4aaf41eeddc34f339adb73b7a8819ddc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-06 05:58:57,138 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:57,138 INFO [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083., storeName=0938af50bcbc9322cb599bce38356083/C, priority=13, startTime=1680760736686; duration=0sec 2023-04-06 05:58:57,138 DEBUG [RS:0;jenkins-hbase20:33239-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-06 05:58:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33239] regionserver.HRegion(9158): Flush requested on 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:57,560 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-06 05:58:57,560 DEBUG [Thread-3384] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f6f7187 to 127.0.0.1:55165 2023-04-06 05:58:57,560 DEBUG [Thread-3384] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:57,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:57,561 DEBUG [Thread-3378] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48502121 to 127.0.0.1:55165 2023-04-06 05:58:57,561 DEBUG [Thread-3378] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:57,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:57,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:57,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:57,566 DEBUG [Thread-3380] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08d3d8bc to 127.0.0.1:55165 2023-04-06 05:58:57,566 DEBUG [Thread-3380] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:57,982 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406a0d7a82a3ef64d1e970e6b7470debc31 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a0d7a82a3ef64d1e970e6b7470debc31 2023-04-06 05:58:58,389 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=729, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/9638b974b55c4eeabfac210ae1e06c42 2023-04-06 05:58:58,644 DEBUG [Thread-3382] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e6ad437 to 127.0.0.1:55165 2023-04-06 05:58:58,645 DEBUG [Thread-3382] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:58,650 DEBUG [Thread-3386] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49c34fbe to 127.0.0.1:55165 2023-04-06 05:58:58,650 DEBUG [Thread-3386] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:58,650 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-06 05:58:58,650 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-06 05:58:58,650 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-06 05:58:58,650 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-06 05:58:58,650 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 356 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5494 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5295 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5346 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5538 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(398): read 5262 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-06 05:58:58,651 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:58:58,651 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6722b207 to 127.0.0.1:55165 2023-04-06 05:58:58,651 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:58:58,652 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-06 05:58:58,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-06 05:58:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-06 05:58:58,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:58:58,656 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760738656"}]},"ts":"1680760738656"} 2023-04-06 05:58:58,657 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-06 05:58:58,669 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-06 05:58:58,670 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, UNASSIGN}] 2023-04-06 05:58:58,671 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, UNASSIGN 2023-04-06 05:58:58,672 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:58:58,672 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760738672"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680760738672"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680760738672"}]},"ts":"1680760738672"} 2023-04-06 05:58:58,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949}] 2023-04-06 05:58:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:58:58,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/73877eca2eca41d29b709cbabc9bcf20 2023-04-06 05:58:58,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=729 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/21c9fab8179f4bbe95800f813d837caf 2023-04-06 05:58:58,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/9638b974b55c4eeabfac210ae1e06c42 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9638b974b55c4eeabfac210ae1e06c42 2023-04-06 05:58:58,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:58,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 0938af50bcbc9322cb599bce38356083, disabling compactions & flushes 2023-04-06 05:58:58,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:58,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9638b974b55c4eeabfac210ae1e06c42, entries=150, sequenceid=729, filesize=25.4 K 2023-04-06 05:58:58,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/73877eca2eca41d29b709cbabc9bcf20 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/73877eca2eca41d29b709cbabc9bcf20 2023-04-06 05:58:58,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/73877eca2eca41d29b709cbabc9bcf20, entries=150, sequenceid=729, filesize=11.9 K 2023-04-06 05:58:58,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/21c9fab8179f4bbe95800f813d837caf as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/21c9fab8179f4bbe95800f813d837caf 2023-04-06 05:58:58,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/21c9fab8179f4bbe95800f813d837caf, entries=150, sequenceid=729, filesize=11.9 K 2023-04-06 05:58:58,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=20.13 KB/20610 for 0938af50bcbc9322cb599bce38356083 in 1281ms, sequenceid=729, compaction requested=false 2023-04-06 05:58:58,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:58,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. after waiting 0 ms 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:58,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 0938af50bcbc9322cb599bce38356083 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=A 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=B 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0938af50bcbc9322cb599bce38356083, store=C 2023-04-06 05:58:58,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-06 05:58:58,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:58:59,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230406ce35cf29f8814d48a018039a80ac53e6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ce35cf29f8814d48a018039a80ac53e6 2023-04-06 05:58:59,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:58:59,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=735, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/0a644b3ab32444088000e7fc32384889 2023-04-06 05:58:59,684 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/11fed6c18ce04fdd96c428c35322c37d 2023-04-06 05:58:59,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/fa5c4bd7ec844e158b941678f94baa49 2023-04-06 05:58:59,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/A/0a644b3ab32444088000e7fc32384889 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0a644b3ab32444088000e7fc32384889 2023-04-06 05:58:59,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0a644b3ab32444088000e7fc32384889, entries=100, sequenceid=735, filesize=18.6 K 2023-04-06 05:58:59,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/B/11fed6c18ce04fdd96c428c35322c37d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/11fed6c18ce04fdd96c428c35322c37d 2023-04-06 05:58:59,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/11fed6c18ce04fdd96c428c35322c37d, entries=100, sequenceid=735, filesize=9.5 K 2023-04-06 05:58:59,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/.tmp/C/fa5c4bd7ec844e158b941678f94baa49 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/fa5c4bd7ec844e158b941678f94baa49 2023-04-06 05:58:59,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/fa5c4bd7ec844e158b941678f94baa49, entries=100, sequenceid=735, filesize=9.5 K 2023-04-06 05:58:59,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 0938af50bcbc9322cb599bce38356083 in 868ms, sequenceid=735, compaction requested=true 2023-04-06 05:58:59,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0aae6df55f764b989cc56f620e9b3bec, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2ff529d2a9b44dddbecdb3991b4c5aa8, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/211abfcbb6454a3ea53e13e3f402fc4b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/824016b66ac3482e9cec6bf3df7ef73f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/b2db5e08fafa4c819eef0babfeaf409e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c82db9bdee1e4941bdf2cd9bc6872ffc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/411c5d5c2f6e4c67a037c4de020e0943, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ca24592675774c9d9f3a99bf5bd3b542, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/aac1f8a4c1b2420597c1fe170a8aef76, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908] to archive 2023-04-06 05:58:59,711 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:59,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c9d74e3083f045cab594882a44af91c2 2023-04-06 05:58:59,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ae5ab87bc788400d93a1e4b357690c95 2023-04-06 05:58:59,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f43363bcd21c4706a0b5aa830f3353ac 2023-04-06 05:58:59,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0aae6df55f764b989cc56f620e9b3bec to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0aae6df55f764b989cc56f620e9b3bec 2023-04-06 05:58:59,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d8f5d26b5ea143b08718d917478aa4b9 2023-04-06 05:58:59,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a9a1be46c84744aaa5a864dab7cc9c21 2023-04-06 05:58:59,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2ff529d2a9b44dddbecdb3991b4c5aa8 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2ff529d2a9b44dddbecdb3991b4c5aa8 2023-04-06 05:58:59,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/7bf96660271c4928aaf34210668b4685 2023-04-06 05:58:59,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ceaf9e7ebf5d417b8edb1348ad01186f 2023-04-06 05:58:59,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/f178abf9b7374584ad776b0afe4081c4 2023-04-06 05:58:59,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/211abfcbb6454a3ea53e13e3f402fc4b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/211abfcbb6454a3ea53e13e3f402fc4b 2023-04-06 05:58:59,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/2710f2a73e2d41b9a62b6d6546451061 2023-04-06 05:58:59,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/824016b66ac3482e9cec6bf3df7ef73f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/824016b66ac3482e9cec6bf3df7ef73f 2023-04-06 05:58:59,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/1cc01e74774348bb8044ce04eddb2fa1 2023-04-06 05:58:59,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5aec9e772bbf473899011f12b604ec36 2023-04-06 05:58:59,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/97f2368254cc4fe3871ccfc59afde0ba 2023-04-06 05:58:59,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/b2db5e08fafa4c819eef0babfeaf409e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/b2db5e08fafa4c819eef0babfeaf409e 2023-04-06 05:58:59,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/4b278b0f27b3450890fdd7e136447e98 2023-04-06 05:58:59,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c82db9bdee1e4941bdf2cd9bc6872ffc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c82db9bdee1e4941bdf2cd9bc6872ffc 2023-04-06 05:58:59,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/c1b1eab666e8410aa852ab27f651d612 2023-04-06 05:58:59,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/a590a59cf13a4212a6de169e2c5f0792 2023-04-06 05:58:59,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/d139149a958f426aa1e12429f63aee9a 2023-04-06 05:58:59,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/411c5d5c2f6e4c67a037c4de020e0943 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/411c5d5c2f6e4c67a037c4de020e0943 2023-04-06 05:58:59,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9ddd4ed5084c4efa8c56da025f43a679 2023-04-06 05:58:59,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/978356407e5b46f29d7e1877cefeb21d 2023-04-06 05:58:59,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ca24592675774c9d9f3a99bf5bd3b542 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/ca24592675774c9d9f3a99bf5bd3b542 2023-04-06 05:58:59,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/e3fd269d7d9b4c3eb6cbb5a7d16577a5 2023-04-06 05:58:59,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/dea6e10912844ff2a60c1458c051b02a 2023-04-06 05:58:59,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/aac1f8a4c1b2420597c1fe170a8aef76 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/aac1f8a4c1b2420597c1fe170a8aef76 2023-04-06 05:58:59,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/5a25de6a65484c4cb2c60d7c4b3ac3de 2023-04-06 05:58:59,738 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/47799ca24a5c4d73b085c7974f432908 2023-04-06 05:58:59,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29ea0e733cce4a06b8769590de1d6d32, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/433061bbfec14e02b822921d2901a759, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f4b9ab6c4ebf4202b55dc996237e2916, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bcecc67fcfa244708108cf0a594bcb10, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d935252a9df34c3fa9f932ee8e9bd6dd, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/3cce81f60e05467e91ebff96f196983d, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/9c4c5ce941284f61bc765f0e5a59d976, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29134fdc14fe421b86c74b547a791d5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5de19d6830da401e8f6a8d4719adcb5f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60] to archive 2023-04-06 05:58:59,743 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:59,745 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d6446c81112541de91c30fe7af83fa87 2023-04-06 05:58:59,746 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/425f018178ed4530aea9a38b1bd4656b 2023-04-06 05:58:59,747 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29ea0e733cce4a06b8769590de1d6d32 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29ea0e733cce4a06b8769590de1d6d32 2023-04-06 05:58:59,748 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/260cb60ec84c442eba620fa6c9f2cd46 2023-04-06 05:58:59,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/0ab9aa3e68764d30a0939c35c036bb0d 2023-04-06 05:58:59,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/433061bbfec14e02b822921d2901a759 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/433061bbfec14e02b822921d2901a759 2023-04-06 05:58:59,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/85ee23875ee74abb8ee5f32303684a90 2023-04-06 05:58:59,751 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/549668857ace437fa1b478c9d976d564 2023-04-06 05:58:59,752 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ffeffc78891742f0a5a5870ab6f105a1 2023-04-06 05:58:59,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f4b9ab6c4ebf4202b55dc996237e2916 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f4b9ab6c4ebf4202b55dc996237e2916 2023-04-06 05:58:59,753 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d86b4746fd3a418ab83e95f5848193c4 2023-04-06 05:58:59,754 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/dce1b9c1b17f4ffabd36cdf9263063ff 2023-04-06 05:58:59,755 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bcecc67fcfa244708108cf0a594bcb10 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bcecc67fcfa244708108cf0a594bcb10 2023-04-06 05:58:59,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/bb4b0e970e014069a0bddb7594c7bc3c 2023-04-06 05:58:59,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/acd8626336d44a72940067897227e5da 2023-04-06 05:58:59,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d935252a9df34c3fa9f932ee8e9bd6dd to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/d935252a9df34c3fa9f932ee8e9bd6dd 2023-04-06 05:58:59,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/90d55dce38504a88ac126e969aa811aa 2023-04-06 05:58:59,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/307975a9dca5409a8fd8cadf8de42af9 2023-04-06 05:58:59,759 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/3cce81f60e05467e91ebff96f196983d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/3cce81f60e05467e91ebff96f196983d 2023-04-06 05:58:59,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/f8041847d8d545beb4f2fb9d27109229 2023-04-06 05:58:59,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/4363e02e665145348af769e9cb66816f 2023-04-06 05:58:59,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/9c4c5ce941284f61bc765f0e5a59d976 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/9c4c5ce941284f61bc765f0e5a59d976 2023-04-06 05:58:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:58:59,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/8b1d35ab22944c80bbc7ef8a51afebd5 2023-04-06 05:58:59,762 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5ab550e8dba44b76bb3c3ef40130a012 2023-04-06 05:58:59,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29134fdc14fe421b86c74b547a791d5f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/29134fdc14fe421b86c74b547a791d5f 2023-04-06 05:58:59,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/ca29240834c4413e86f095cd31e63aa2 2023-04-06 05:58:59,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/adc7d42e475947428a6f49f1b1da1a66 2023-04-06 05:58:59,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5de19d6830da401e8f6a8d4719adcb5f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/5de19d6830da401e8f6a8d4719adcb5f 2023-04-06 05:58:59,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/597bd7260556474f8cc39ba7d7faae50 2023-04-06 05:58:59,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/e764eff398da4f35a2acca44ec7cc500 2023-04-06 05:58:59,768 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/aba5990e18a8468197a8b9776a2b6f60 2023-04-06 05:58:59,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/2fee635788084055a39b9287a990d9f1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ee633458de3746f78006b1aaa6b2193c, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7fdae584c1c24a09b289f9c1a4adff57, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0ded247bd174440abea2101f5fea999, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1c4735ad4f064a21bec218f1a4f4a744, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/9d899e2bec9242a4b420d9f6d8d46454, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ab04ceb1865a49848e801fee448f9831, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0d55711a46a4ce58a7d86408b317430, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94324eddddf446cd89d1c65a023884f6, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684] to archive 2023-04-06 05:58:59,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-06 05:58:59,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ec2f551a6c4f46d59cffb04536bbde8f 2023-04-06 05:58:59,772 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7f53666d64fd4b19acde65962af3bb26 2023-04-06 05:58:59,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/2fee635788084055a39b9287a990d9f1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/2fee635788084055a39b9287a990d9f1 2023-04-06 05:58:59,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a35b8af07ff54aabb71d149322ac0d6c 2023-04-06 05:58:59,775 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/163c4c6982934b9281d39675b3e41403 2023-04-06 05:58:59,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ee633458de3746f78006b1aaa6b2193c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ee633458de3746f78006b1aaa6b2193c 2023-04-06 05:58:59,777 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d04ada10781d47c8abe706b1bd2d809f 2023-04-06 05:58:59,778 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/36f575f7abe74dfda466bd28c3a8f4e9 2023-04-06 05:58:59,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/109feeb580084a1797d88c100deda0cc 2023-04-06 05:58:59,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7fdae584c1c24a09b289f9c1a4adff57 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/7fdae584c1c24a09b289f9c1a4adff57 2023-04-06 05:58:59,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/f5bd6d1a8c87429dbb3d3630ae3c2b02 2023-04-06 05:58:59,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e7f717d987434274814193e202ffc99e 2023-04-06 05:58:59,783 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0ded247bd174440abea2101f5fea999 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0ded247bd174440abea2101f5fea999 2023-04-06 05:58:59,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/23f713dd0fec4948b42db6d3835d7152 2023-04-06 05:58:59,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e00bdce19eed4c63ab647c877af062f0 2023-04-06 05:58:59,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1c4735ad4f064a21bec218f1a4f4a744 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1c4735ad4f064a21bec218f1a4f4a744 2023-04-06 05:58:59,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b1bd5eabc8c14b55a4abed5ddaf7b019 2023-04-06 05:58:59,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/1969c974df6e44599c8294259e4694d1 2023-04-06 05:58:59,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/9d899e2bec9242a4b420d9f6d8d46454 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/9d899e2bec9242a4b420d9f6d8d46454 2023-04-06 05:58:59,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/b497004a2c2742b8ad1ac038d901d693 2023-04-06 05:58:59,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94d84094059a48b29cbb3d63acacdc4e 2023-04-06 05:58:59,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ab04ceb1865a49848e801fee448f9831 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/ab04ceb1865a49848e801fee448f9831 2023-04-06 05:58:59,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/d4ed9970fdf948919a7f06d4b0c69e27 2023-04-06 05:58:59,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/e6d429dd440f4137ae7217566540a054 2023-04-06 05:58:59,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0d55711a46a4ce58a7d86408b317430 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/c0d55711a46a4ce58a7d86408b317430 2023-04-06 05:58:59,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/bad6ef92c1ef4677bc29bbd2a815cff9 2023-04-06 05:58:59,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/70bdf78c631247f39a0dc9ff25a1bf9e 2023-04-06 05:58:59,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94324eddddf446cd89d1c65a023884f6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/94324eddddf446cd89d1c65a023884f6 2023-04-06 05:58:59,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/38522ca9c7d84512b91f390fba6729ee 2023-04-06 05:58:59,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/97f00f0d633c47de91d8d63dfb3ea224 2023-04-06 05:58:59,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/a896c173f4e34787ba70a27f84375684 2023-04-06 05:58:59,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits/738.seqid, newMaxSeqId=738, maxSeqId=4 2023-04-06 05:58:59,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083. 2023-04-06 05:58:59,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 0938af50bcbc9322cb599bce38356083: 2023-04-06 05:58:59,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 0938af50bcbc9322cb599bce38356083 2023-04-06 05:58:59,817 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=0938af50bcbc9322cb599bce38356083, regionState=CLOSED 2023-04-06 05:58:59,817 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680760739817"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680760739817"}]},"ts":"1680760739817"} 2023-04-06 05:58:59,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-06 05:58:59,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 0938af50bcbc9322cb599bce38356083, server=jenkins-hbase20.apache.org,33239,1680760570949 in 1.1450 sec 2023-04-06 05:58:59,822 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-06 05:58:59,822 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0938af50bcbc9322cb599bce38356083, UNASSIGN in 1.1500 sec 2023-04-06 05:58:59,823 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680760739823"}]},"ts":"1680760739823"} 2023-04-06 05:58:59,824 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-06 05:58:59,826 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-06 05:58:59,828 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-06 05:59:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-06 05:59:00,763 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-06 05:59:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-06 05:59:00,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,768 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,769 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 05:59:00,780 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:59:00,783 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C, FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits] 2023-04-06 05:59:00,787 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0a644b3ab32444088000e7fc32384889 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/0a644b3ab32444088000e7fc32384889 2023-04-06 05:59:00,788 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/646393a34eab40539e75c4dd23ba629f to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/646393a34eab40539e75c4dd23ba629f 2023-04-06 05:59:00,791 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9638b974b55c4eeabfac210ae1e06c42 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/A/9638b974b55c4eeabfac210ae1e06c42 2023-04-06 05:59:00,795 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/11fed6c18ce04fdd96c428c35322c37d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/11fed6c18ce04fdd96c428c35322c37d 2023-04-06 05:59:00,798 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/73877eca2eca41d29b709cbabc9bcf20 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/73877eca2eca41d29b709cbabc9bcf20 2023-04-06 05:59:00,800 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/78d44b2b01f24a4cba77f335cf91b16d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/B/78d44b2b01f24a4cba77f335cf91b16d 2023-04-06 05:59:00,803 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/21c9fab8179f4bbe95800f813d837caf to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/21c9fab8179f4bbe95800f813d837caf 2023-04-06 05:59:00,805 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/4aaf41eeddc34f339adb73b7a8819ddc to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/4aaf41eeddc34f339adb73b7a8819ddc 2023-04-06 05:59:00,807 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/fa5c4bd7ec844e158b941678f94baa49 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/C/fa5c4bd7ec844e158b941678f94baa49 2023-04-06 05:59:00,810 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits/738.seqid to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083/recovered.edits/738.seqid 2023-04-06 05:59:00,811 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/.tmp/data/default/TestAcidGuarantees/0938af50bcbc9322cb599bce38356083 2023-04-06 05:59:00,811 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-06 05:59:00,811 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:59:00,812 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-06 05:59:00,817 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ec0420f6d5943bfb228b55a5fd684c0 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304060ec0420f6d5943bfb228b55a5fd684c0 2023-04-06 05:59:00,819 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040611730eb17d1047b29d2551919e9910f7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040611730eb17d1047b29d2551919e9910f7 2023-04-06 05:59:00,820 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040614233e8fce4e409d905bbe667ae2b44a to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040614233e8fce4e409d905bbe667ae2b44a 2023-04-06 05:59:00,822 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b40786055ed4220a163a0a915c75234 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304061b40786055ed4220a163a0a915c75234 2023-04-06 05:59:00,824 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062ba214d963fd452f867ace94455a20a1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304062ba214d963fd452f867ace94455a20a1 2023-04-06 05:59:00,825 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406430b8f385f934151b93bbccbb1a8f4b3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406430b8f385f934151b93bbccbb1a8f4b3 2023-04-06 05:59:00,827 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e7ceca4f94142889b37d6ef3e031fe2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304064e7ceca4f94142889b37d6ef3e031fe2 2023-04-06 05:59:00,829 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fab0a5776214c0c815aec303164881d to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fab0a5776214c0c815aec303164881d 2023-04-06 05:59:00,830 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fdef8e0eaf5471e8e079db23c133408 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304065fdef8e0eaf5471e8e079db23c133408 2023-04-06 05:59:00,832 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661fa911d35704fbab50fd118433b164c to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040661fa911d35704fbab50fd118433b164c 2023-04-06 05:59:00,834 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066356407a9d8447e1a39446f8779d08f1 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066356407a9d8447e1a39446f8779d08f1 2023-04-06 05:59:00,836 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040663d9db0e75014ec3be03d8d0ff723cb2 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040663d9db0e75014ec3be03d8d0ff723cb2 2023-04-06 05:59:00,837 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066c0579f1e1be44709c2074a0eb4bb968 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304066c0579f1e1be44709c2074a0eb4bb968 2023-04-06 05:59:00,839 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406770a2180a29c4bd98d74f1e89f1b9b42 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406770a2180a29c4bd98d74f1e89f1b9b42 2023-04-06 05:59:00,840 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069aa1c4d7c1f642f7a50bcd76961b11ee to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304069aa1c4d7c1f642f7a50bcd76961b11ee 2023-04-06 05:59:00,842 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a01f7e7656c34d9786d4a3dfc32d699e to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a01f7e7656c34d9786d4a3dfc32d699e 2023-04-06 05:59:00,843 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a0d7a82a3ef64d1e970e6b7470debc31 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406a0d7a82a3ef64d1e970e6b7470debc31 2023-04-06 05:59:00,845 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406afa3ef52ff3a47b0a2fd9ebb6f1d47f4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406afa3ef52ff3a47b0a2fd9ebb6f1d47f4 2023-04-06 05:59:00,846 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b7e6757a72254dcdb6711f01fada54b3 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406b7e6757a72254dcdb6711f01fada54b3 2023-04-06 05:59:00,847 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c65ddb026255420b849fe82b95dc8ee7 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c65ddb026255420b849fe82b95dc8ee7 2023-04-06 05:59:00,848 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6bc66e4af0c47bfad46f2030bf1b7ce to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406c6bc66e4af0c47bfad46f2030bf1b7ce 2023-04-06 05:59:00,849 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ce35cf29f8814d48a018039a80ac53e6 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406ce35cf29f8814d48a018039a80ac53e6 2023-04-06 05:59:00,850 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e66f5944525e453188dad0d22f21866b to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406e66f5944525e453188dad0d22f21866b 2023-04-06 05:59:00,851 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f2d73f2d6e114d4ebec9ae778769a5b4 to hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230406f2d73f2d6e114d4ebec9ae778769a5b4 2023-04-06 05:59:00,851 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-06 05:59:00,853 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,854 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-06 05:59:00,856 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-06 05:59:00,856 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,857 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-06 05:59:00,857 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680760740857"}]},"ts":"9223372036854775807"} 2023-04-06 05:59:00,858 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-06 05:59:00,858 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0938af50bcbc9322cb599bce38356083, NAME => 'TestAcidGuarantees,,1680760713870.0938af50bcbc9322cb599bce38356083.', STARTKEY => '', ENDKEY => ''}] 2023-04-06 05:59:00,858 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-06 05:59:00,858 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680760740858"}]},"ts":"9223372036854775807"} 2023-04-06 05:59:00,860 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-06 05:59:00,861 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-06 05:59:00,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 97 msec 2023-04-06 05:59:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35907] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-06 05:59:00,871 INFO [Listener at localhost.localdomain/33939] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-06 05:59:00,883 INFO [Listener at localhost.localdomain/33939] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229 (was 229), OpenFileDescriptor=571 (was 538) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=412 (was 421), ProcessCount=167 (was 167), AvailableMemoryMB=4290 (was 4283) - AvailableMemoryMB LEAK? - 2023-04-06 05:59:00,884 INFO [Listener at localhost.localdomain/33939] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-06 05:59:00,885 INFO [Listener at localhost.localdomain/33939] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-06 05:59:00,885 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1545112e to 127.0.0.1:55165 2023-04-06 05:59:00,885 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:59:00,885 DEBUG [Listener at localhost.localdomain/33939] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-06 05:59:00,885 DEBUG [Listener at localhost.localdomain/33939] util.JVMClusterUtil(257): Found active master hash=818052816, stopped=false 2023-04-06 05:59:00,885 INFO [Listener at localhost.localdomain/33939] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:59:00,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 05:59:00,886 INFO [Listener at localhost.localdomain/33939] procedure2.ProcedureExecutor(629): Stopping 2023-04-06 05:59:00,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-06 05:59:00,886 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:59:00,887 DEBUG [Listener at localhost.localdomain/33939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e4f328e to 127.0.0.1:55165 2023-04-06 05:59:00,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 05:59:00,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-06 05:59:00,887 DEBUG [Listener at localhost.localdomain/33939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:59:00,887 INFO [Listener at localhost.localdomain/33939] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,33239,1680760570949' ***** 2023-04-06 05:59:00,887 INFO [Listener at localhost.localdomain/33939] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-06 05:59:00,888 INFO [RS:0;jenkins-hbase20:33239] regionserver.HeapMemoryManager(220): Stopping 2023-04-06 05:59:00,888 INFO [RS:0;jenkins-hbase20:33239] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-06 05:59:00,888 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-06 05:59:00,888 INFO [RS:0;jenkins-hbase20:33239] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-06 05:59:00,888 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(3304): Received CLOSE for c823e58660018648a7124b90ca124d5e 2023-04-06 05:59:00,888 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:59:00,888 DEBUG [RS:0;jenkins-hbase20:33239] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30d140d4 to 127.0.0.1:55165 2023-04-06 05:59:00,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c823e58660018648a7124b90ca124d5e, disabling compactions & flushes 2023-04-06 05:59:00,889 DEBUG [RS:0;jenkins-hbase20:33239] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:59:00,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:59:00,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:59:00,889 INFO [RS:0;jenkins-hbase20:33239] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-06 05:59:00,889 INFO [RS:0;jenkins-hbase20:33239] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-06 05:59:00,889 INFO [RS:0;jenkins-hbase20:33239] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-06 05:59:00,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. after waiting 0 ms 2023-04-06 05:59:00,889 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-06 05:59:00,889 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:59:00,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c823e58660018648a7124b90ca124d5e 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-06 05:59:00,890 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-06 05:59:00,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-06 05:59:00,890 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1479): Online Regions={c823e58660018648a7124b90ca124d5e=hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e., 1588230740=hbase:meta,,1.1588230740} 2023-04-06 05:59:00,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-06 05:59:00,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-06 05:59:00,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-06 05:59:00,890 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-06 05:59:00,890 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-06 05:59:00,892 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1505): Waiting on 1588230740, c823e58660018648a7124b90ca124d5e 2023-04-06 05:59:00,896 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/info/61f9852ec8514ffbadc2009a968ce872 2023-04-06 05:59:00,900 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61f9852ec8514ffbadc2009a968ce872 2023-04-06 05:59:00,977 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-06 05:59:01,092 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1505): Waiting on 1588230740, c823e58660018648a7124b90ca124d5e 2023-04-06 05:59:01,172 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-06 05:59:01,172 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-06 05:59:01,293 DEBUG [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1505): Waiting on 1588230740, c823e58660018648a7124b90ca124d5e 2023-04-06 05:59:01,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/.tmp/info/510569bfe44840e19a797472a42aa10e 2023-04-06 05:59:01,306 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/rep_barrier/3cc896b211574389831ed1edcd8b05dd 2023-04-06 05:59:01,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/.tmp/info/510569bfe44840e19a797472a42aa10e as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/info/510569bfe44840e19a797472a42aa10e 2023-04-06 05:59:01,312 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cc896b211574389831ed1edcd8b05dd 2023-04-06 05:59:01,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/info/510569bfe44840e19a797472a42aa10e, entries=2, sequenceid=6, filesize=4.8 K 2023-04-06 05:59:01,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c823e58660018648a7124b90ca124d5e in 425ms, sequenceid=6, compaction requested=false 2023-04-06 05:59:01,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/namespace/c823e58660018648a7124b90ca124d5e/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-06 05:59:01,319 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/table/750e3c9bb62544f7b4fd47f08228ea4d 2023-04-06 05:59:01,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:59:01,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c823e58660018648a7124b90ca124d5e: 2023-04-06 05:59:01,320 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680760573466.c823e58660018648a7124b90ca124d5e. 2023-04-06 05:59:01,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 750e3c9bb62544f7b4fd47f08228ea4d 2023-04-06 05:59:01,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/info/61f9852ec8514ffbadc2009a968ce872 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/info/61f9852ec8514ffbadc2009a968ce872 2023-04-06 05:59:01,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 61f9852ec8514ffbadc2009a968ce872 2023-04-06 05:59:01,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/info/61f9852ec8514ffbadc2009a968ce872, entries=22, sequenceid=93, filesize=7.1 K 2023-04-06 05:59:01,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/rep_barrier/3cc896b211574389831ed1edcd8b05dd as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/rep_barrier/3cc896b211574389831ed1edcd8b05dd 2023-04-06 05:59:01,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cc896b211574389831ed1edcd8b05dd 2023-04-06 05:59:01,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/rep_barrier/3cc896b211574389831ed1edcd8b05dd, entries=6, sequenceid=93, filesize=5.5 K 2023-04-06 05:59:01,331 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/.tmp/table/750e3c9bb62544f7b4fd47f08228ea4d as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/table/750e3c9bb62544f7b4fd47f08228ea4d 2023-04-06 05:59:01,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 750e3c9bb62544f7b4fd47f08228ea4d 2023-04-06 05:59:01,334 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/table/750e3c9bb62544f7b4fd47f08228ea4d, entries=9, sequenceid=93, filesize=5.5 K 2023-04-06 05:59:01,335 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 445ms, sequenceid=93, compaction requested=false 2023-04-06 05:59:01,339 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-06 05:59:01,340 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-06 05:59:01,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-06 05:59:01,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-06 05:59:01,341 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-06 05:59:01,493 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,33239,1680760570949; all regions closed. 2023-04-06 05:59:01,506 DEBUG [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/oldWALs 2023-04-06 05:59:01,507 INFO [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C33239%2C1680760570949.meta:.meta(num 1680760573246) 2023-04-06 05:59:01,515 DEBUG [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/oldWALs 2023-04-06 05:59:01,515 INFO [RS:0;jenkins-hbase20:33239] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C33239%2C1680760570949:(num 1680760572253) 2023-04-06 05:59:01,515 DEBUG [RS:0;jenkins-hbase20:33239] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:59:01,515 INFO [RS:0;jenkins-hbase20:33239] regionserver.LeaseManager(133): Closed leases 2023-04-06 05:59:01,516 INFO [RS:0;jenkins-hbase20:33239] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-06 05:59:01,516 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 05:59:01,519 INFO [RS:0;jenkins-hbase20:33239] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:33239 2023-04-06 05:59:01,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 05:59:01,527 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,33239,1680760570949 2023-04-06 05:59:01,527 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5a9138df rejected from java.util.concurrent.ThreadPoolExecutor@5eaea755[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2602] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:01,528 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-06 05:59:01,528 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7979f250 rejected from java.util.concurrent.ThreadPoolExecutor@5eaea755[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2602] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:01,528 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,33239,1680760570949] 2023-04-06 05:59:01,529 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,33239,1680760570949; numProcessing=1 2023-04-06 05:59:01,530 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,33239,1680760570949 already deleted, retry=false 2023-04-06 05:59:01,530 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,33239,1680760570949 expired; onlineServers=0 2023-04-06 05:59:01,530 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,35907,1680760569837' ***** 2023-04-06 05:59:01,530 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-06 05:59:01,530 DEBUG [M:0;jenkins-hbase20:35907] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71ff74eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-06 05:59:01,530 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,35907,1680760569837 2023-04-06 05:59:01,530 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,35907,1680760569837; all regions closed. 2023-04-06 05:59:01,530 DEBUG [M:0;jenkins-hbase20:35907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-06 05:59:01,530 DEBUG [M:0;jenkins-hbase20:35907] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-06 05:59:01,531 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-06 05:59:01,531 DEBUG [M:0;jenkins-hbase20:35907] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-06 05:59:01,531 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680760571962] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680760571962,5,FailOnTimeoutGroup] 2023-04-06 05:59:01,531 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680760571957] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680760571957,5,FailOnTimeoutGroup] 2023-04-06 05:59:01,531 INFO [M:0;jenkins-hbase20:35907] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-06 05:59:01,531 INFO [M:0;jenkins-hbase20:35907] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-06 05:59:01,531 INFO [M:0;jenkins-hbase20:35907] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-06 05:59:01,531 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-06 05:59:01,531 DEBUG [M:0;jenkins-hbase20:35907] master.HMaster(1512): Stopping service threads 2023-04-06 05:59:01,532 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-06 05:59:01,532 INFO [M:0;jenkins-hbase20:35907] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-06 05:59:01,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-06 05:59:01,532 ERROR [M:0;jenkins-hbase20:35907] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (562943890) connection to localhost.localdomain/127.0.0.1:32913 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-06 05:59:01,533 INFO [M:0;jenkins-hbase20:35907] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-06 05:59:01,533 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-06 05:59:01,533 DEBUG [M:0;jenkins-hbase20:35907] zookeeper.ZKUtil(398): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-06 05:59:01,533 WARN [M:0;jenkins-hbase20:35907] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-06 05:59:01,533 INFO [M:0;jenkins-hbase20:35907] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-06 05:59:01,533 INFO [M:0;jenkins-hbase20:35907] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-06 05:59:01,534 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-06 05:59:01,534 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:59:01,534 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:59:01,534 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-06 05:59:01,534 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:59:01,534 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.30 KB heapSize=393.94 KB 2023-04-06 05:59:01,630 INFO [RS:0;jenkins-hbase20:33239] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,33239,1680760570949; zookeeper connection closed. 2023-04-06 05:59:01,630 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 05:59:01,630 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@142f475 rejected from java.util.concurrent.ThreadPoolExecutor@5eaea755[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2602] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:01,631 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1673cc2f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1673cc2f 2023-04-06 05:59:01,631 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): regionserver:33239-0x10084d914bb0001, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 05:59:01,632 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4dde5bf9 rejected from java.util.concurrent.ThreadPoolExecutor@5eaea755[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2602] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:01,633 INFO [Listener at localhost.localdomain/33939] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-06 05:59:01,945 INFO [M:0;jenkins-hbase20:35907] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.30 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c3340f30e1454bf0a8a707993bfb5fd4 2023-04-06 05:59:01,948 INFO [M:0;jenkins-hbase20:35907] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c3340f30e1454bf0a8a707993bfb5fd4 2023-04-06 05:59:01,949 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c3340f30e1454bf0a8a707993bfb5fd4 as hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c3340f30e1454bf0a8a707993bfb5fd4 2023-04-06 05:59:01,952 INFO [M:0;jenkins-hbase20:35907] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c3340f30e1454bf0a8a707993bfb5fd4 2023-04-06 05:59:01,952 INFO [M:0;jenkins-hbase20:35907] regionserver.HStore(1080): Added hdfs://localhost.localdomain:32913/user/jenkins/test-data/9ca0827d-7877-3195-59b0-4ee96e3850f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c3340f30e1454bf0a8a707993bfb5fd4, entries=65, sequenceid=578, filesize=24.7 K 2023-04-06 05:59:01,953 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegion(2948): Finished flush of dataSize ~345.30 KB/353585, heapSize ~393.92 KB/403376, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-06 05:59:01,953 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-06 05:59:01,953 DEBUG [M:0;jenkins-hbase20:35907] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-06 05:59:01,956 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-06 05:59:01,956 INFO [M:0;jenkins-hbase20:35907] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-06 05:59:01,957 INFO [M:0;jenkins-hbase20:35907] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:35907 2023-04-06 05:59:01,958 DEBUG [M:0;jenkins-hbase20:35907] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,35907,1680760569837 already deleted, retry=false 2023-04-06 05:59:02,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 05:59:02,060 INFO [M:0;jenkins-hbase20:35907] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,35907,1680760569837; zookeeper connection closed. 2023-04-06 05:59:02,060 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@743fbab4 rejected from java.util.concurrent.ThreadPoolExecutor@652e2a7a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4349] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:02,060 DEBUG [Listener at localhost.localdomain/33939-EventThread] zookeeper.ZKWatcher(600): master:35907-0x10084d914bb0000, quorum=127.0.0.1:55165, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-06 05:59:02,060 ERROR [Listener at localhost.localdomain/33939-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63a0d466 rejected from java.util.concurrent.ThreadPoolExecutor@652e2a7a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4349] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-06 05:59:02,064 WARN [Listener at localhost.localdomain/33939] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-06 05:59:02,069 INFO [Listener at localhost.localdomain/33939] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-06 05:59:02,180 WARN [BP-1364566030-148.251.75.209-1680760566858 heartbeating to localhost.localdomain/127.0.0.1:32913] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-06 05:59:02,181 WARN [BP-1364566030-148.251.75.209-1680760566858 heartbeating to localhost.localdomain/127.0.0.1:32913] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1364566030-148.251.75.209-1680760566858 (Datanode Uuid aceb380d-48cd-4fee-9534-415073b880c6) service to localhost.localdomain/127.0.0.1:32913 2023-04-06 05:59:02,183 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962/dfs/data/data1/current/BP-1364566030-148.251.75.209-1680760566858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 05:59:02,184 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9b523263-a3f1-a568-54ea-203388963099/cluster_8238166b-fe50-b19f-8e35-2a4b3769a962/dfs/data/data2/current/BP-1364566030-148.251.75.209-1680760566858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-06 05:59:02,214 INFO [Listener at localhost.localdomain/33939] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-06 05:59:02,373 INFO [Listener at localhost.localdomain/33939] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-06 05:59:02,510 INFO [Listener at localhost.localdomain/33939] hbase.HBaseTestingUtility(1293): Minicluster is down